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FOREWORD 


This  development  effort  was  conducted  under  Contract  N00123-79-C-1335  with 
Anacapa  Sciences,  Inc.  in  support  of  Navy  Decision  Coordinating  Paper  Z11/(PPN, 
subproject  Z117Q-PN.05.  Reducing  Manpower  Costs  Through  Better  System  Design.  It  was" 
sponsored  by  the  Beputy  Chief  of  Naval  Operations  (Manpower,  Personnel,  and  Training, 
OP-01). 

The  objective  of  the  subproject  is  to  develop  techniques  for  analysis  of 
hardware/software/personnel  trade-offs  at  all  stages  of  system  design.  The  objective  of 
this  effort  was  to  conduct  an  evaluation  of  "An  Engineer's  Guide  to  the  Use  of  Human 
Resources  in  Electronic  Systems  Design,"  which  has  been  developed  to  assist  hardware 
developers  to  reduce  the  number  and  skills  of  operators  and  maintainers  required  by  Navy 
surface  ship  electronic  systems. 

Many  talented,  dedicated  people  within  the  Navy  Department  and  contractor  design 
and  development  companies  contributed  to  the  evaluation.  Particular  appreciation  is 
extended  to  individuals  from  the  following  organizations,  who  provided  substantive 
comments  and  guidance  during  the  project. 

•  Chief  of  Naval  Education  and  Training 

•  Naval  Electronics  Systems  Command 

•  Naval  Sea  Systems  Command 

•  Naval  Weapons  Engineering  Support  Activity 

•  Naval  Electronics  Systems  Security  Engineering  Center 

•  Naval  Ocean  Systems  Center 

•  Chief  of  Naval  Operations 

•  Bendix,  Electrodynamics  Division 

•  FMC,  Northern  Ordnance  Division 

•  General  Electric,  Electronics  Systems  Division 

•  Goodyear,  Aerospace-Defense  Systems  Division 

•  Gould,  Chesapeake  Instrument  Division 

•  Hughes  Aircraft,  Ground  Systems  Group 

•  ITT-Gilfillan 

•  Litton,  Datalog  Division 

•  Lockheed  Electronics,  Systems  Division 

•  Lockheed,  Aircraft-Missiles  and  Space  Division 

•  Radio  Corporation  of  America,  Missiles  and  Surface  Radar  Division 

•  Singer,  Librascope  Division 

•  Sperry,  Microwave  Electronics 


JAMES  F.  KELLY,  JR. 
Commanding  Officer 


JAMES  J.  REGAN 
Technical  Director 
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SUMMARY 


Problem 


The  Navy  cannot  provide  sufficient  numbers  of  adequately  skilled  operators  and 
maintainers  for  sophisticated  shipboard  electronic  systems.  One  of  the  steps  that  can  be 
taken  toward  the  solution  of  this  problem  is  to  reduce  the  human  resource  requirements  of 
new  system  acquisitions.  In  a  previous  project,  An  Engineer's  Guide  to  the  Use  of  Human 
Resources  in  Electronics  Systems  Design  was  developed  to  help  control  the  system 
acquisition  and  design  process.  The  guide  provided  both  data  and  direction  to  reduce 
manning  requirements,  starting  with  design  decisions  made  at  the  earliest  stages  of 
conceptual  development. 

Objective 

The  objective  of  this  effort  was  to  evaluate  the  Engineer's  Guide  to  determine  the 
validity  of  its  methodology,  the  adequacy  of  its  data,  and  the  degree  to  which  it  can  be 
implemented.  A  preliminary  objective  was  to  identify  specific  users  who  would  apply  the 
Engineer's  Guide  in  the  systems  design  process. 

Approach 

A  survey  was  administered  to  members  of  the  Navy  systems  acquisition  and 
development  communities  and  to  a  representative  sample  of  the  senior  engineering  design 
and  management  community  within  contractor  facilities.  Respondents  were  asked  to 
evaluate  the  usefulness  of  each  major  section  of  the  guide. 

Rating  data  were  gathered  and  analyzed  on  several  utility  and  adequacy  factors. 
Also,  comments  were  gathered  and  analyzed,  and  important  themes  were  extracted. 

Results 


The  guide's  overall  potential  utility  was  clearly  evident  from  the  results.  However, 
many  specific  suggestions  were  received  for  improving  specific  sections  of  the  guide  and 
for  refining  its  overall  format. 

Conclusions 


•  The  guide  will  have  a  positive  effect  on  future  Navy  electronic  systems  by 
making  human  resources  a  specific  design  consideration  and  by  providing  necessary 
technical  data. 

•  The  guide  must  be  issued  in  the  form  of  a  military  instruction,  handbook,  or 
standard  if  it  is  to  be  implemented  effectively . 

•  The  guide  needs  to  be  reorganized  to  facilitate  access  to  data.  Additional 
information  should  be  added  to  several  sections. 

•  Specifications  must  be  developed  to  direct  implementation  of  the  guide  by  the 
Navy  systems  acquisition  and  development  community  and  application  by  the  contractor 
design  and  development  community. 


v 


Recommendations 


•  Revision  of  the  guide  should  include  direct  inputs  from  technical  experts  in  the 
Navy  acquisition  and  contractor  design  communities. 

•  The  process  of  making  the  guide  an  official  military  document  (instruction, 
handbook,  or  standard)  should  begin. 

•  The  guide  must  be  issued  in  a  format  (hardbound  copy,  three-ring  notebook, 
computer  data  bank,  etc.)  that  will  allow  cost-effective  updating,  because  much  of  the 
most  useful  material  is  time-sensitive.  Alternative  media  should  be  explored. 
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INTRODUCTION 


Problem  and  Background 


For  several  years,  the  Navy  has  been  unable  to  provide  sufficient  numbers  of 
adequately  skilled  operators  and  maintainers  for  sophisticated  shipboard  electronic 
systems.  Manpower  problems  in  operating  and  maintaining  these  systems  are  widely 
recognized  in  the  Navy  and  in  the  Office  of  the  Secretary  of  Defense.  To  deal  with  these 
problems,  the  Assistant  Secretary  of  Defense  (Manpower,  Reserve  Affairs,  and  Logistics) 
has  outlined  new  constraining  requirements  for  dealing  with  manning  support  problems 
before  final  decisions  are  made  to  acquire  new  major  systems.?  Also,  the  Navy  has  begun 
a  program  to  document  and,  eventually,  monitor  and  control  the  problems  in  matching 
manpower  resources  with  new  system  hardware  requirements.  The  intent  of  this  program, 
which  was  called  the  HARDMAN  Program,  was  to  develop  a  regulatory  structure  to 
ensure  that  manpower  requirements  would  become  a  major  factor  in  making  system 
acquisition  and  design  decisions.^' 

Sh-  o  support  ASD  requirements  and  complement  the  HARDMAN  regulatory  structure, 
the  Navy  Personnel  Research  and  Development  Center  began  development  of  the 
Engineer's  Guide  to  the  Use  of  Human  Resources  in  Electronics  System  Design.^,  The 
purpose  of  the  guide  was  to  provide  a  data  base  that  would  help  the  Navy  acquisition  and 
development  community  and  the  contractor  design  community  to  develop  electronic 
systems  that  require  fewer  human  resources — in  terms  of  both  numbers  and  skill  levels.  It 
was  a  first  attempt  at  assembling  existing  data,  gathering  new  pertinent  data,  and 
packaging  the  data  in  a  document  that  could  be  called  out  by  system  acquisition  and 
development  organizations  and  implemented  by  design  engineers. 


The  guide  was  limited  in  scope,  covering  five  types  of  electronic  systems  aboard 
surface  combatants:  sonar,  radar,  fire  control,  communications,  and  data  processing.  It 
included  an  introductory  chapter,  a  chapter  on  designing  in  relation  to  human  resources, 
and  pertinent  data  under  the  following  sections: 

1.  Definition  and  Impact  of  Design  Concepts.  This  section  included  a  list  of  21 
design  concepts  (e.g.,  built-in  test  equipment,  standard  hardware  components,  automatic 
decision  making). 


2.  Interaction  of  Design  Concept  Impacts  on  Different  System  Design  Criteria. 
This  section  included  a  list  of  14  system  design  criteria  (e.g.,  total  number  of  operators 
required,  initial  system  acquisition  costs,  shipboard  maintenance  man-hours  required),  and 
results  obtained  when  a  group  of  Navy  experts  in  engineering,  acquisition,  and 
maintenance  rated  each  of  the  21  design  concepts  against  each  of  the  14  system  design 
criteria.1' 


‘ASD  Memo  of  17  August  1978. 

2 Military  Manpower  vs.  Hardware  Procurement  Study  (HARDMAN),  (Final  Report). 
Washington,  DC:  Chief  of  Naval  Operations,  October  1977. 

3  An  Engineer's  Guide  to  the  Use  of  Human  Resources  in  Electronic  Systems  Design 
(NPRDC  Tech.  Note  79-8).  San  Diego:  Navy  Personnel  Research  and  Development 
Center,  December  1978. 

*See  Tables  1  and  2  in  the  following  section  (pp.  20  and  36)  for  complete  listings  of 
the  design  concepts  and  criteria. 


3.  Types  of  Technicians  Assigned  to  Surface  Ship  Electronic  Systems.  This  section 
identified  the  types  of  technicians  assigned  to,  and  their  responsibilities  for,  operation  and 
maintenance  of  the  five  types  of  electronic  systems  addressed  in  the  guide. 

4.  Projected  Supply  of  Technical  Ratings  at  Different  Experience  Levels.  There  is 
a  projected  shortfall  of  experienced  technicians  of  virtually  every  type  associated  with 
surface  ship  electronic  systems.  Specific  shortages  were  presented  in  this  section. 

5.  Evaluation  of  Alternatives.  This  section  provided  aids  which  the  designers  could 
use  to  conduct  tradeoff  analyses  involving  manning,  skills,  and  other  design  criteria  for 
alternative  hardware  systems. 

6.  Taxonomies  of  Tasks  and  Associated  Skill  Levels.  This  section  presented  data  on 
the  capabilities  of  first-,  second-,  and  third-class  petty  officers  in  performing  operational 
and  maintenance  tasks  required  by  the  five  types  of  electronics  systems  covered  in  the 
guide.  These  data  were  intended  to  help  designers  avoid  tasks  that  were  too  difficult  or 
time-consuming  for  technicians  at  lower  skill  levels. 

7.  Different  and  Time-consuming  Tasks.  This  was  a  listing  of  tasks,  selected  from 
6  above,  that  were  either  difficult  to  perform  (skill  intensive)  or  required  excessive  time 
to  accomplish  (manpower  intensive). 

8.  Training  Requirements  and  Navy  Enlisted  Classifications  (NECs).  This  was  a 
listing  of  the  job  specialties  and  training  requirements  for  the  technicians  listed  in  Section 

3. 


9.  Billet  Life  Cycle  Costs  for  Required  Personnel.  This  section  included  data  for 
estimating  personnel  life  cycle  costs  for  selected  system  life  cycles  (1,  5,  10,  15,  and  20 
years).5 


10.  References. 

Objective 

The  objective  of  this  effort  was  to  evaluate  the  technical  data  and  methods  included  in 
the  guide  before  implementing  them  in  the  acquisition  and  design  communities. 
Specifically,  answers  were  sought  to  the  following  questions: 

1.  Does  the  guide  provide  a  reasonable  and  feasible  concept  for  reducing  the 
current  and  projected  manning  problems  with  the  Navy's  shipboard  electronic  systems? 

2.  What  specific  user  communities  can  apply  the  guide  effectively  during  system 
design  and  acquisition? 

3.  Is  the  specific  approach  taken  in  the  guide  viable? 

4.  Are  the  specific  data  and  procedures  presented  in  the  guide  accurate,  complete, 
understandable,  and  usable? 

5.  What  revisions  are  required  in  the  guide  to  assure  its  successful  implementation? 


5See  Koehler,  E.  A.  Manpower  Availability-- Navy  Enlisted  Projections--FY78-84 
(NPRDC  Spec.  Rep.  79-1).  San  Diego:  Navy  Personnel  Research  and  Development 
Center,  1979. 
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METHOD 


User  Identification 


Acquisition/Development  Community 

The  guide  was  distributed  by  the  Chief  of  Naval  Materiel  (Assistant  Deputy  Chief 
(Technology  and  Laboratories))  to  a  variety  of  naval  commands  and  organizations  within 
the  Department  of  Defense  (DoD)  as  well  as  to  individuals  who  had  provided  assistance  in 
developing  the  guide  or  had  expressed  an  interest  in  using  it. 

To  obtain  preliminary  reactions  and  to  solicit  aid  in  identifying  potential  users, 
approximately  100  people  within  DoD  were  contacted  by  telephone  about  6  weeks  after 
the  guide  had  been  distributed.  In  the  majority  of  cases,  persons  contacted  either  had  not 
received  the  guide  or  had  not  had  an  opportunity  to  review  it.  For  those  who  had  not 
received  the  guide,  arrangements  were  made  to  deliver  another  copy,  along  with  an 
Information  Feedback  Sheet,  which  was  intended  to  obtain  information  on  the  recipient's 
own  qualifications,  experience  with  system  development  programs,  knowledge  of  programs 
under  development  that  could  benefit  from  the  guide,  specific  individuals  who  might  be 
users  of  the  guide,  and  reactions  to  a  series  of  questions  concerning  the  feasibility  of 
implementing  the  guide  and  the  completeness  of  its  content.  For  those  who  had  received 
and  reviewed  the  guide,  telephone  interviewers  solicited  preliminary  reactions  and 
suggestions  as  to  potential  users.  These  potential  users  were  contacted  and  arrangements 
made  to  provide  a  copy  of  the  guide  and  the  Information  Feedback  Sheet  to  them. 

To  obtain  a  detailed  critique  of  the  guide,  detailed  comments  on  the  best  users  and 
likely  utility  of  the  guide,  :md  specific  suggestions  regarding  its  introduction  and 
implementation  in  the  acquisition  community,  group  interviews  were  conducted  at  The 
Naval  Sea  Systems  Command  (NAVSEA)  (Codes  SEA-05L1C,  SEA- 3134,  and  PMS- 30612), 
the  Naval  Electronics  Systems  Command  (NAVELEX)  (Codes  ELEX-4701  and  5203),  the 
Naval  Electronics  Systems  Security  Engineering  Center,  and  Naval  Ocean  Systems 
Command  (NOSC)  (Codes  924  and  921).  A  semistructured  interview  technique  was  used, 
in  which  a  standard  set  of  questions  and  topics  was  used  to  guide  the  discussion  and  ensure 
that  all  pertinent  topics  were  covered.  However,  interviewees  were  allowed  to  digress 
from  specific  topics  or  expand  the  coverage  of  a  topic  to  explore  areas  that  were  not 
anticipated  during  development  of  the  structured  questions.  An  Evaluation  Debriefing 
Form,  which  included  the  questions  and  topics,  as  well  as  space  for  recording  inter¬ 
viewee's  comments,  was  used  for  these  interviews. 

Originally,  plans  had  been  to  conduct  the  evaluation  entirely  within  the  Navy 
acquisition  community.  After  the  telephone  interviews  and  the  group  interviews  were 
completed,  the  guide  was  to  be  presented  to  a  sample  of  acquisition  groups  that  had  Navy 
electronic  systems  under  development.  Members  of  these  groups  were  to  evaluate  the 
guide's  applicability  over  a  period  of  2  to  3  months  and  then  participate  in  group 
debriefings.  The  Evaluation  Debriefing  Form  was  to  be  used  again  in  these  sessions. 

However,  results  obtained  from  the  Information  Feedback  Sheets  and  the  first  round 
of  group  interviews  showed  that  it  was  not  feasible  to  test  the  guide  using  only  members 
of  the  acquisition  community.  The  guide's  technical  application  to  system  design  was  seen 
as  almost  totally  the  responsibility  of  the  contractor  design  community.  Further,  it  was 
felt  that  the  guide  would  have  to  be  suitably  reformatted,  as  a  military  instruction, 
handbook,  or  standard,  before  it  could  be  implemented  through  contractual  specification. 
For  these  reasons,  it  was  decided  to  shift  primary  emphasis  of  the  evaluation  to  the 
contractor  design  community. 


3 


Contractor  Design  Community 

Lockheed's  DIALOG  search  system  was  used  to  search  the  Frost  and  Sullivan  Defense 

2 

Market  Measure  System  (DM  )  data  base  to  identify  contractors  with  then-current  design 
and  development  programs  in  the  five  areas  covered  by  the  guide.  The  search  parameters 
specified:  Navy  contracts,  electronic  systems,  stage  of  work  to  be  either  R&D  or 
production,  and  a  minimum  program  size  of  $5  million  or  more  for  R<5cD  or  $20  million  or 
more  for  production.  The  cost  criteria  were  included  to  ensure  that  the  programs  were 
major  contracts  within,  at  least,  Acquisition  Category  (AC AT)  III,  and  represented  the  full 
Weapons  System  Acquisition  Process.6 

Approximately  20  development  programs  in  sonar,  radar,  fire  control,  communica¬ 
tions,  or  data  processing  were  identified  that  met  the  above  criteria.  For  each  program, 
the  computer  search  provided  descriptive  summaries,  which  included  the  sponsoring 
agency,  contracting  office,  and  contract  number.  Contract  numbers  were  used  to 
identify  the  technical  point  of  contact  within  each  sponsoring  agency,  through 
assistance  from  the  contracting  offices.  These  contacts  were  then  telephoned  and 
briefed  on  the  guide's  purpose  and  scope  and  the  need  to  survey  high-level  designers 
and  design  managers  in  the  contractor  community.  In  almost  all  cases,  the  technical 
point  of  contact  for  each  program  identified  several  individuals  in  the  contractor's 
facility  who  would  be  qualified  to  review  and  comment  on  the  guide. 

Next,  the  chief  contact  at  each  contractor  facility  was  contacted  by  a  letter,  which 
briefly  explained  the  purpose  and  scope  of  the  guide,  the  history  of  problems  in  operating 
and  maintaining  sophisticated  electronic  equipment  in  the  Navy,  the  regulatory  structure 
being  created  under  the  HARDMAN  program,  the  procedures  for  the  survey,  and  the 
required  qualifications  of  participants.  Because  it  was  believed  that  most  participants 
would  review  the  guide  and  fill  out  the  survey  instrument  on  their  own  time,  the  letter 
stated  that  a  $100  honorarium  would  be  given  to  each  respondent. 

Follow-up  phone  calls  were  made  to  identify  those  chief  contacts  that  wanted  their 
groups  to  participate  in  the  survey  and  to  identify  qualified  individuals  within  each  group. 
Respondents  had  to  have  substantial  design  experience  in  one  or  more  of  the  five  areas  of 
electronics  systems  covered  by  the  guide  and  had  to  be  a  program  manager,  project 
engineer,  or  a  senior  member  of  the  design  staff.  Because  of  the  close  relationship 
between  fire  control  systems  and  launcher  and  gun  mount  systems,  designers  with 
experience  in  the  latter  were  also  included.  Initially,  52  qualified  respondents  from  1 3 
contractor  facilities  agreed  to  participate  in  the  survey.  Of  these,  10  dropped  out  due  to 
work  schedule  conflicts. 

Survey  Instruments 

Two  survey  instruments  were  developed--Evaluation  Package  C,  for  the  contractor 
community,  and  Evaluation  Package  G,  for  the  DoD  acquisition  community.  The  two 
instruments  were  identical,  except  that  Evaluation  Package  G  did  not  mention  the  $100 
honorarium,  which  could  not  be  offered  to  DoD  personnel,  and  addressed  the  respondents 
in  the  context  of  acquisition  instead  of  design. 


®Many  ACAT  III  programs  are  made  up  of  multiple  contracts,  all  of  which  may  be 
smaller  than  specified  in  the  cost  criteria.  Thus,  the  cost  criteria  served  to  exclude  these 
programs. 
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Respondents  were  asked  to  evaluate  the  usefulness  of  the  information  in  the  guide's 
introductory  chapters  and  major  sections  by  responding  to  six  questions,  using  a  five-point 
response  scale,  where  1  =  the  most  positive,  and  5  =  the  most  negative.  These  questions, 
along  with  their  anchor  scales,  are  listed  below: 

1.  Considering  my  present  job  as  a  designer,  the  information  in _ is  (Very 

useful — Completely  useless). 

2.  If  DoD  were  to  seriously  impose  design  constraints  based  on  manpower  con¬ 
siderations,  the  information  in _ would  be  (Very  useful — Completely  useless). 

3.  The  information  in _ appears  to  be  (Very  accurate — Very  question- 

aole). 

4.  The  information  in _ is  (Thorough  and  complete — Full  of  gaps  and 

omissions). 

5.  The  format  (tables,  charts,  text,  etc.)  for  presenting  the  key  information 

in _ is  (Appropriate:  Easy  to  understand — Inappropriate:  Hard  to  understand). 

6.  In  order  to  provide  helpful  guidance  to  designers  like  me, _ should  be 

(Kept  exactly  as  is — Changed  drastically  (or  omitted). 

Respondents  who  rated  parts  of  the  guide  as  either  4  or  5  were  asked  to  provide 
comments  as  to  the  reasons  for  their  negative  ratings.  If  they  were  not  familiar  with  the 
information  in  a  particular  part  of  the  guide,  they  were  asked  to  check  a  "Don't  know"  box 
provided. 

Also,  respondents  were  asked  (1)  to  indicate  whether  they  understood  the  21  design 
concepts  (Section  1)  and  the  14  system  design  criteria  (Section  2)  and  whether  they  felt 
they  were  useful  as  is,  useful  if  modified,  or  useless,  (2)  to  explain  why  they  felt  concepts 
or  criteria  should  be  modified  or  dropped,  and  (3)  to  describe  any  concepts  or  criteria  that 
they  felt  should  be  added  to  the  list. 

Finally,  respondents  were  asked  to  provide  information  as  to  their  job  title,  type  of 
job,  experience  with  specific  systems,  and  amount  of  time  spent  in  examining  the  guide 
and  filling  out  the  survey  form. 

Evaluation  Package  C  was  sent  to  the  42  respondents  from  contractor  facilities  who 
agreed  to  participate  in  the  survey;  and  Evaluation  Package  G,  to  members  of  the  Navy 
acquisition  community  who  had  provided  unusually  detailed  critiques  of  the  guide  earlier. 
Appendix  A  provides  a  copy  of  Evaluation  Package  C. 

All  of  the  Evaluation  Package  C  instruments  and  three  of  the  Evaluation  Package  G 
instruments  were  returned.  Since  there  were  no  systematic  differences  in  the  data  from 
the  two  packages,  the  45  sets  of  data  were  pooled  for  statistical  analysis. 

Data  Analysis 

Rating  Data  from  Evaluation  Packages 

A  frequency  analysis  was  performed  on  the  rating  data  from  the  45  respondents  to 
Evaluation  Packages  C  and  G.  Histograms  were  generated  to  show  the  distribution  of 
responses  to  the  six  questions  asked  concerning  the  usefulness  of  the  guide's  preliminary 


chapters  and  major  sections.  Also,  percentages  of  responses  as  to  the  understandability 
and  usefulness  of  the  21  design  concepts  and  the  14  system  design  criteria  were  derived. 

Comments  from  All  Evaluators 


Comments  on  the  guide  were  recorded  in  a  variety  of  forms.  This  included  notes 
made  in  Evaluation  Packages  C  and  G,  telephone  interviews,  group  interviews,  Informa¬ 
tion  Feedback  Sheets,  letters,  and  memos.  All  comments  were  analyzed  to  identify  and 
extract  every  "complete  comment"  (i.e.,  a  comment  that  stood  by  itself  as  a  complete, 
intelligible  utterance  concerning  a  topic  related  to  the  guide).  From  this  analysis,  1,084 
complete  comments  were  identified,  and  each  was  transcribed  onto  a  separate  card  along 
with  several  lines  of  identifying  code  (see  Figure  1).  In  transcribing,  comments  were 
categorized  according  to  their  nature  and  content.  Usually  categories  were  the  titles  of 
major  sections  of  the  guide.  Within  categories,  specific  topics  were  identified.  Topics 
were  usually  one  of  the  scales,  or  structured  questions,  within  a  survey  instrument. 

An  analyst  then  synthesized  all  of  the  comments  within  each  topic  in  an  attempt  to 
remove  some  of  the  redundancy  among  comments  and  to  recast  comments  so  that 
thoughts  flowed  in  a  logical  and  comprehensible  fashion. 

Appendix  B  lists  DoD  and  contractor  personnel  who  made  substantive  contributions  to 
the  evaluation  of  the  Engineer's  Guide. 


Figure  l.  Example  comment  card  with  key  to  identifiers. 
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RESULTS 


Respondents  to  Evaluation  Packages  C  and  G  spent  an  average  ol  5.5  hours  in 
reviewing  the  guide,  with  a  standard  deviation  of  3.4  hours;  and  an  average  of  3.3  hours  in 
filling  out  the  evaluation  packages,  with  a  standard  deviation  of  2.0  hours.  Since  these 
respondents  provided  the  majority  of  comments  included  in  this  section,  many  remarks  are 
critical,  because  respondents  were  specifically  requested  to  explain  why  they  gave 
negative  ratings.  However,  the  overall  majority  of  respondents  generally  reacted 
favorably  to  the  guide,  as  is  evident  from  the  summarized  rating  data. 

In  reorganizing  many  of  the  comments  to  follow  the  sequence  of  topics  in  the  guide, 
it  was  necessary  to  lift  short  sections  of  longer  comments  out  of  context.  To  reestablish 
a  commentator's  context  and  meaning,  it  was  often  necessary  to  add  or  delete  a  few 
words  or  to  restructure  some  phrases.  Because  the  synthesis  of  comments  from  many 
sources  into  a  reasonably  coherent  set  of  statements  on  a  given  topic  in  the  guide  did  not 
usually  preserve  the  exact  original  wording  of  each  commentator,  quotation  marks  are  not 
generally  used  in  the  following  material.  Also,  it  should  be  noted  that: 

1.  A  colon  is  used  to  indicate  the  end  of  prefatory  remarks  by  the  analyst. 

2.  A  paragraph  generally  contains  closely  related  notions. 

3.  Where  several  respondents'  statements  are  contained  in  a  paragraph,  all  of  the 
comments  by  a  single  respondent  are  in  a  single  sentence,  concluded  by  a  period. 
Semicolons  and  dashes  are  used  to  punctuate  complete  thoughts  within  each  respondent's 
sentence. 


4.  Where  a  single  respondent’s  statements  constitute  a  paragraph,  several  sentences 
may  be  used  to  express  several  thoughts.  The  context  unambiguously  distinguishes 
between  single-respondent  and  multiple-respondent  paragraphs. 

5.  The  number  in  parentheses  behind  each  question/ topic  indicates  how  many  of  the 
45  respondents  answered  or  addressed  that  particular  question/topic. 

Chapters  1  and  lit  Introductory  Material 

There  was  little  enthusiasm  for  the  material  presented  in  Chapters  1  and  II  of  the 
guide.  Although  most  respondents  considered  that  the  information  was  of  some  use,  it 
apparently  did  not  provide  an  effective  introduction.  The  problem  was  not  so  much  the 
accuracy  of  what  was  presented,  but  certain  gaps  and  omissions.  Suggested  changes  and 
additions  included  the  following  primary  themes:  rewrite  the  existing  material, 
demonstrate  application  of  data  from  the  guide  to  the  design  process,  and  add  certain 
technical  issues  (such  as  equating  system  performance  requirements  to 
maintenance/operator  requirements,  other  system  design  tradeoffs,  and  a  weighting 
system  for  concept  evaluation). 
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Data  from  Six  Questions  in  Evaluation  Packages 

Figure  2  shows  how  respondents  answered  questions  regarding  the  information  in  the 
first  two  chapters  of  the  guide.  Comments  relative  to  these  questions  are  provided  below. 

1.  Useful  for  Present  Job?  (15) 

Those  who  believed  that  the  introductory  material  was  useful  said:  It  could 
provide  information  to  people  who  were  unfamiliar  with  the  topic.  Describes  an  area  that 
few  design  engineers  have  considered.  Provides  general  awareness. 

Those  who  believed  that  it  was  not  useful  said:  Training,  manpower  costs, 
availability,  and  Navy-imposed  constraints  are  beyond  the  scope  of  the  designer.  Informa¬ 
tion  was  much  too  complicated  and  detailed  for  any  program  manager  or  designer  to  take 
seriously.  No  direct  practical  application. 

Other  comments  included:  The  information  might  provide  a  nice  background  and 
awareness  of  the  problem  but  only  a  shallow  treatment  of  its  symptoms  and  possible 
causes.  The  subject  is  well  known  to  a  depth  much  greater  than  this  elementary  overview. 
It  provides  essential  information  on  background  and  objectives,  but  it  provides  no 
operational  direction— does  not  tell  the  designer  what  to  DO. 

2.  Useful  for  Serious  Manpower  Constraints?  (14) 

Here  the  comments  focused  more  on  the  realities  of  trying  to  apply  the  data  in 
the  introduction.  On  the  positive  side,  statements  included:  Programs  will  initially  be 
more  expensive,  though  life  cycle  costs  would  probably  be  reduced.  Material  needs  to  be 
expanded— what  facilities  are  involved,  how  does  the  Navy  system  work,  what  are  the 
details  of  organizational  versus  depot  repair,  etc.? 

On  the  negative  side,  comments  included:  The  information  is  of  little  use 
because  it  is  generalized  and  may  not  be  correct  or  appropriate  for  a  specific  system. 
The  data  are  basically  motherhood  and  do  not  fully  address  the  total  system  design 
problem;  minimizing  the  manpower  requirement,  taken  to  its  ultimate,  will  result  in  a 
completely  unaffordable  system  in  terms  of  hardware  cost  and  development  cost. 

Other  comments  were:  The  material  provides  interesting  questions  and  insights 
but  misdirects  attention  from  the  real  problem.  The  information  is  instructive,  but  is  not 
definitive  nor  provided  in  a  manner  that  could  make  it  obligatory  to  the  designer. 

On  the  latter  theme,  others  observed:  Specific  requirements  and  cost  goals 
would  need  to  be  defined.  Navy  constraints  need  to  be  identified.  In  general,  a  statement 
of  goals  like  that  provided  in  these  chapters  does  not  tell  the  designer  what  to  DO. 

One  commentator  stated:  Designers  are  always  under  the  gun  in  the  following 
order:  function,  schedule,  and  cost. 

3.  Accurate?  (6) 

Comments  included:  Information  is  true  and  accurate,  but  only  to  awaken  the 
uninitiated,  not  the  experienced  systems  engineer.  The  data  are  general  and  nonspecific. 
The  information  should  be  expanded. 

One  commentator  observed:  1  never  experienced  a  DoD  customer  that  was 
serious  about  maintainability. 
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Figure  2.  Ratings  of  information  in  Chapters!  and  II: 
Introductory  Material. 


4.  Complete?  (14) 

On  the  positive  side,  comments  included:  The  information  is  informative  and 
interesting.  Information  is  general  and  not  exhaustive,  but  probably  adequate  for 
introducing  the  concept  of  the  guide. 

On  the  negative  side,  statements  included:  Doesn't  make  sense  until  the 
remainder  of  the  book  is  read.  Difficult  to  extract  the  true  intent  of  this  document.  Full 
of  redundancy,  especially  in  the  four  factors  driving  manpower  tasks;  nowhere  was 
expressed  the  basic  concept  that  all  systems  have  two  major  elements,  performance  and 
cost,  and  that  the  intent  of  this  document  is  to  cut  operating  and  life  cycle  costs  without 
sacrificing  performance.  Very  general.  Does  not  address  the  complete  system  design 
problem  and  would  result  in  a  biased  design. 

Other  respondents  observed:  The  maintenance  philosophy  varies  widely  with  the 
system  type  and  end  use  (e.g.,  aircraft  versus  submarine).  Need  better  definitions; 
acronyms  need  to  be  spelled  out. 

One  commentator  suggested  that  it  would  really  help  the  designer  if  insight  were 
provided  into  the  shipboard  organizational  problem  and  the  difficulties  associated  with 
maintenance  task  execution. 

5.  Understandable?  (12) 

All  the  comments  indicated  that  the  material  in  Chapters  I  and  II  is  difficult  to 
understand.  Typical  comments  were:  Terms  are  undefined  and  the  figures  too  complex— 
it  is  very  difficult  to  understand  what  points  are  being  made.  Too  much  detail  in  graphics 
without  supporting  text.  Too  many  words  in  boxes- -too  hard  to  read;  not  familiar  with  all 
terms  used.  Print  too  small;  too  much  information;  too  complicated  to  be  readily 
understood;  designers  simply  won't  bother  with  this  kind  of  material.  Charts  could  be 
presented/discussed  more  clearly. 

Comments  on  specific  figures  include:  Figure  2  requires  more  explanation. 
Figure  2  not  needed.  Figure  1— explain  the  acronyms  for  the  uninitiated,  e.g.,  MPT&S, 
DSARC,  etc.  Figure  3  is  too  complicated  to  be  readable. 

One  respondent  suggested  that  the  charts  be  subdivided  according  to  the  user- 
charts  for  Navy  management  should  be  separate  from  those  charts  which  are  directed  to 
the  design  engineer. 

One  commentator  observed  that  in  large  programs,  objectives  for  human 
resources  are  stated  much  earlier  than  indicated  in  Figure  2:  Waiting  to  apply  "Navy- 
imposed  criteria"  until  after  Step  5  of  Figure  3  is  too  late. 

6.  Should  be  Changed?  (21) 

Comments  indicate  that  this  material  should  be  changed  and  improved  rather 
than  eliminated.  One  commentator  remarked  that  the  orientation  provided  by  this 
material  is  important  and  should  be  expanded  so  that  the  remainder  of  the  book  is  not 
used  by  rote.  Suggestions  for  changing  the  manner  in  which  the  information  is  presented 
included:  Rewrite  completely;  1  had  to  read  most  sentences  three  or  four  times  before  I 
could  understand  them.  Define  acronyms  for  those  not  familiar  with  them;  a  table  of 
definitions  would  be  useful. 
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A  large  number  of  comments  focused  on  the  need  to  demonstrate  the  application  of 
this  information  to  the  design  process:  The  information  is  of  such  a  general  nature  that 
the  imposition  of  these  data,  by  contract,  on  the  designer  will  not  really  result  in  specific 
design  action.  Does  not  provide  guidance,  only  information.  Expand  and  show  examples; 
provide  design  review  guidelines.  Provide  more  concrete  information  on  the 
symptoms/impacts  of  the  manpower  resources  being  mismatched  with  systems;  the 
concepts  and  milestones  in  Figure  1  are  not  explained  fully  enough  to  be  useful.  I  could 
not  find  a  smooth  transitional  flow  from  the  premise  to  conclusion;  the  beginning  must  be 
extremely  clear  and  describe  precisely  the  intent  of  the  document,  the  problems,  the 
approach  and  the  benefits.  Need  more  information  on  how  this  would  be  implemented  and 
how  the  design  will  be  constrained. 

A  number  of  comments  focused  on  specific  technical  issues:  Expand  information  to 
equate  system  performance  requirements  to  maintenance/operator  training  requirements. 
Expand  to  at  least  address  other  areas  of  system  design  tradeoffs;  indicate  a  weighting 
system  to  help  in  concept  evaluation.  Rewrite  in  one  cohesive  form  (basic  ideas  are  good) 
and  add  information  that  different  systems  will  have  different  responses  to  the  questions 
rather  than  leave  this  an  implied  point. 

One  respondent  noted  an  apparent  lack  of  direction  to  a  specific  user  audience  for 
the  guide.  He  stated  that  Chapters  1  and  II  seem  to  be  a  rationale  for  a  book  written  for 
Navy  personnel  rather  than  design  engineers. 

Another  commentator  pointed  out  that  consideration  must  be  given  to  include  data  on 
the  Standard  Electronic  Module  Program  (SEMP)  and  Navy  Standard  Processors  (e.g., 
AN/UYK-7/20,  AN/UYQ-21,  etc.),  which  are  often  specified  as  required  equipments  and 
which  have  significant  impact  on  designers. 

One  commentator  pinpointed  the  real  issues  behind  the  guide.  He  stated:  The  last  20 
years  of  DoD  and  NASA  history  are  replete  with  attempts  to  impose  expensive, 
complicated  solutions  to  similar  problems;  designers  and  program  managers  are  usually 
constrained  by  "design  to  cost"  or  similar  features,  and  if  human  resources  are  to  be 
considered,  a  simpler,  more  direct  index  must  be  provided. 

Comments  from  Other  Sources 


The  following  general  comments  came  from  sources  other  than  Evaluation  Packages 
C  and  G.  To  a  certain  extent,  these  comments  are  similar  to  comments  covered  under  the 
various  questions  listed  above. 

Several  commentators  stated:  The  flow  chart  in  Figure  3  was  useful;  however,  the 
top  and  bottom  halves  of  the  flow  chart  should  be  connected  because  the  arrow  didn't 
seem  to  go  anywhere.  Also,  the  connection  between  the  flow  chart  and  the  boxes  that 
precede  each  of  the  following  sections  in  the  guide  should  be  highlighted  to  help  the 
reader  make  the  connection  between  the  master  flow  chart  in  Figure  3  and  the  remainder 
of  the  guide. 

There  were  several  comments  to  the  effect  that  the  introductory  material  was 
confusing  at  first  but  understandable  after  a  reading  of  the  guide.  This  would  indicate 
that  the  introductory  material  was  counterproductive. 

One  respondent  wrote:  The  material  was  too  academic;  the  design  engineer  does  not 
think  in  those  terms.  The  questions  posed  in  the  introductory  material  could  be  retained 
in  the  form  of  a  checklist,  but  they  should  be  made  simpler.  A  different  general 
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observation  was;  The  introductory  material  should  be  beefed  up  to  include  more  about  the 
methodology  and  potential  application  of  the  report.  Another  commentator  wrote:  The 
title  and  introductory  material  give  an  impression  of  promising  to  solve  more  problems 
than  the  guide  actually  does;  the  guide  claims  it  provides  a  more  rigorous,  objective  and 
tight  methodology  than  is,  in  fact,  presented. 

Several  commentators  stated  that  the  best  audience  and  users  of  the  guide  would  be 
manpower  and  training  analysts  within  the  Navy.  These  commentators  stated:  Personnel 
skill  level  and  availability  information  already  exists  (contrary  to  a  statement  in  the 
introductory  material  in  the  guide),  but  it  is  not  used  by  designers.  The  manual  could  be 
best  used  as  a  "monitoring"  tool  to  review  contractor  proposals.  Contractor  designers 
need  to  be  influenced  in  order  to  impact  human  resources  requirements  through  system 
design,  but  this  would  be  best  accomplished  by  using  the  guide  to  monitor  contractors 
rather  than  by  developing  a  procedure  for  having  contractors  implement  the  guide 
themselves. 

One  evaluator  included  a  general  comment  about  the  graphics  used  in  the  guide.  He 
believed  that  most  of  the  figures  needed  redesign  to  increase  eye  appeal  and  to  highlight 
the  central  point  illustrated  by  each  figure. 

Finally,  one  evaluator  seemed  to  want  to  absolve  designers  of  responsibility  in 
addressing  Navy  human  resources  problems.  He  stated:  The  responsibility  must  be  placed 
on  the  individual  maintenance  man  to  maintain  a  certain  level  of  proficiency  in  order  to 
advance. 

Section  1:  Definition  and  Impact  of  Design  Concepts 

Section  1  was  rated  as  being  both  useful  and  understandable.  Respondents  com¬ 
mented  that  the  section  would  have  been  even  more  useful  had  greater  detail  been 
provided,  particularly  on  the  impact  of  design  concepts  on  system  cost.  Also,  there  was 
concern  about  the  difficulty  in  applying  information  of  this  type  to  the  design  of  future 
systems. 

As  shown  in  Figure  3,  the  accuracy  of  the  data  were  questioned.  Concerns  included 
the  judgmental  nature  of  the  data,  the  relatively  small  sample  of  judges,  the  qualifica¬ 
tions  of  the  judges,  the  lack  of  validation,  and  the  lack  of  information  on  response 
variability. 

Twenty-six  respondents  recommended  that  this  section  be  changed  in  some  manner. 
Principal  recommendations  were  to:  reorganize  and  reformat  the  material  for  easier 
application,  improve  the  measurement  scale,  and  expand  the  number  of  judges  and  the 
scope  of  their  experience. 

Data  from  Six  Questions  in  Evaluation  Packages 

Figure  3  shows  how  respondents  answered  questions  regarding  the  information  in 
Section  1.  Comments  relative  to  these  questions  are  provided  below. 

1.  Useful  for  Present  Job?  (16) 

Several  positive  comments  included:  This  is  an  excellent  presentation.  Table  1 
and  the  definition  of  the  21  design  concepts  are  useful.  Useful  in  creating  awareness  of 
operator/maintainer  tradeoff  philosophy.  Principal  payoff  is  in  large  systems  (e.g., 
AEGIS,  DD  963,  FFG,  DDGX). 
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Figure  3.  Ratings  of  information  in  Section  1:  Definition 
and  Impact  of  Design  Concepts. 
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Two  commentators  saw  the  usefulness  of  the  data  and  made  the  following 
suggestions  for  improvements:  A  more  detailed  discussion  of  the  significant  elements  of 
Table  1  would  be  helpful;  why,  for  example,  does  "automatic  decision-making"  have  such  a 
negative  impact  on  maintenance?  The  information  presented  is  quite  informative,  but  is 
difficult  to  assimilate;  the  data  shown  in  the  charts  would  be  more  effective  if  some 
summary  and  cross-correlation  were  provided. 

Several  respondents  expressed  difficulty  in  accepting  judgment  data:  Provide 
the  data  point  for  consideration,  but  the  measurable  utility  is  questionable;  the  21 
questions  are  not  really  all  pertinent  and  the  data  used  to  measure  them  are  opinions,  not 
hard  facts.  Don't  really  know  what  the  percent  beneficial/adverse  impact  means  or  what 
it  is  based  on;  evidently  it  is  totally  subjective.  As  stated,  the  data  are  subjective.  A 
larger  data  base  may  provide  different  results  and  would  be  desirable. 

Several  comments  demonstrated  concern  for  relating  the  data  to  cost  considera¬ 
tions:  It's  hard  to  determine  the  value  of  the  data;  how  does  one  use  this  in  terms  of 
dollar  tradeoffs  in  the  system  design  process;  what  is  the  appropriate  algorithm?  I  could 
not  relate  the  data  to  system  costs.  The  buying  activity  should  indicate  the  importance  of 
these  things  prior  to  proposal  evaluation  and  negotiation,  not  use  this  approach  as  a  fudge 
factor  after  the  fact  to  justify  a  decision;  the  cost  impact  of  this  approach  would  be 
serious.  Two  commentators  indicated  the  information  misses  the  mark  in  system  design: 
For  a  system  design  concept,  much  of  the  information  is  irrelevant.  All  programs, 
regardless  of  concept,  already  specify  use  of  good  equipment  layout,  standard  hardware, 
operational  simplicity,  use  of  LRUs,  etc.  Considering  the  magnitude  of  the  problem,  I  do 
not  feel  that  many  of  the  design  concepts  are  viable,  particularly  if  an  effort  is  made  to 
categorize  hardware  (e.g.,  simple/complex,  electronic/mechanical,  onshore/shipboard, 
etc.). 

One  respondent  had  a  series  of  objections  to  the  approach:  The  whole  concept 
and  the  basic  premise  of  quantifiable  design  concepts  for  tradeoff  studies  are  faulty.  The 
areas  which  are  listed  in  Section  1  are  not  congruent  (e.g.,  automatic  decision-aid  is  much 
more  complex  and  dependent  upon  implementation  than  are  the  four  approaches  concern¬ 
ing  LRUs).  The  designer  does  not  have  flexibility  in  all  of  these  areas.  There  are  many 
more  areas  that  have  not  been  considered— use  of  standard  building  blocks,  backfit  vs. 
forwardfit  interfaces,  and  modularity,  reaction  time,  performance  vs.  cost,  etc.  The  data 
used  to  quantify  the  concepts  are  based  on  a  limited  set  of  opinions.  Several  concepts  are 
too  general  to  be  useful,  while  others  are  misrepresented.  In  many  cases,  the  utility  and 
impact  will  vary  between  systems  and  specific  implementations. 


Several  comments  echoed  those  that  were  made  under  the  topic  above  and  will 
not  be  repeated.  Two  comments  suggested  that  the  approach  would  be  useful:  The 
information  is  useful  for  trends.  The  approach  would  be  useful  with  realistic,  quantitative 
data. 

Two  commentators  indicated  that  the  approach  was  not  new:  DoD  is  already 
imposing  restrictions  on  certain  new  ship  and  systems  development  (e.g.,  AEGIS,  CG  47); 
requirements  have  been  established  since  the  contract  start  in  1970.  This  approach  has 
little  use;  there  is  nothing  really  new  here. 

One  respondent  said  that  manpower  should  be  responsive  to  design,  not  the 
reverse:  This  is  a  cart-before-the-horse  approach  based  on  a  single  constraint- 

manpower.  It's  like  squeezing  a  balloon— manpower  can  be  brought  down,  but  other  items, 
like  system  cost,  will  go  up.  It's  not  practical. 


One  comment  implied  difficulty  in  implementing  the  approach  via  contract:  The 
data  are  not  presented  in  a  manner  than  can  be  quantified  and  dictated  by  specification 
parameters. 

One  commentator  suggested  that  the  information  could  be  updated  to  get  a 
perspective  on  the  design  concepts  from  the  operator/maintainer  viewpoint. 

Two  comments  indicated  that  the  data  are  complicated  and  cumbersome:  There 
seem  to  be  too  many  choices;  this  serves  to  complicate  the  design  process.  There  is  no 
handy  way  to  find  the  required  data;  it  would  be  better  to  relate  it  to  a  specific  design 
under  consideration. 

Another  comment  concerns  additional  difficulties  in  applying  the  approach:  The 
21  design  concepts  are  not  all  under  designer  control  and  many  address  issues  that  are 
greater  than  the  single  system,  e.g.,  platform/force  maintenance  problems  and  concepts. 

3.  Accurate?  (16) 

One  respondent  stated:  In  most  cases,  the  data  agree  with  my  intuition  and 
experience.  Another  commentator  took  the  opposite  view:  I  have  considerable  disagree¬ 
ment  with  the  conclusions  drawn  on  the  effect  of  the  design  concepts  on  the  A  through  N 
System  Design  Criteria. 

A  large  number  of  comments  focused  on  the  accuracy  of  the  data  base: 
Accuracy  of  all  tables  in  all  sections  is  questionable;  variance  is  not  given,  but  it  is  stated 
that  there  are  wide  ranges.  There  was  no  presentation  of  any  data  to  validate  the 
impacts.  There  was  not  enough  information  on  the  way  the  data  were  obtained  or  the 
qualifications  of  the  respondents;  also,  the  number  of  respondents  seems  low.  I  have  to  be 
a  little  skeptical  of  the  limited  number  of  raters;  were  they  balanced  with  regard  to  their 
respective  backgrounds  and  professional  specialties?  There  is  not  enough  information  on 
the  32  judges  to  trust  their  judgment  over  my  personal  experience  and  that  of  the  people 
who  work  for  me.  The  attempt  to  quantify  the  impacts  of  these  design  concepts  is 
dangerous,  since  it  is  based  on  a  small  set  of  opinions;  in  some  cases,  the  concept  appears 
to  have  been  misrepresented  or  misunderstood;  in  others,  the  impact  is  so  dependent  on 
specific  implementation  that  a  generalization  is  ridiculous  (e.g.,  automated  support  of 
operations  and  maintenance).  The  sample  size  is  small  and  biased  in  favor  of  the 
procuring  agency's  viewpoint;  increase  the  sample  size  by  using  industrial  equipment 
designers  (not  prime  contractors);  comparing  a  "design  concept"  to  a  "baseline  concept" 
may  give  a  perceived  advantage  or  disadvantage  but  little  accurate  quantitative  data. 

Some  commentators  disputed  the  accuracy  of  data  presented  for  specific  design 
concepts:  When  system  complexity  is  increased,  MTBF  is  increased,  not  decreased  as 
stated  in  the  guide.  Data  are  at  least  debatable;  for  example,  LRU  concepts  are 
dependent  on  the  size  of  the  LRU;  a  limit  of  five  lowest  replaceable  units,  as  compared  to 
20  makes  a  major  difference  in  the  weights.  1  had  trouble  with  the  view  on  embedded 
computers  and  with  some  of  the  analyses  on  initial  acquisition  cost;  also,  the  analysis  of 
supply  and  support  costs  appear  to  support  the  Navy  myth  rather  than  actual  experience 
costs. 


One  respondent  described  how  the  accuracy  of  the  data  could  be  improved  by 
using  a  different  system  for  categorizing  the  design  concepts:  Design  concepts  cover  too 
broad  a  range.  They  could  be  better  categorized  into  small  systems  or  units,  subsystems, 
large  systems,  unmanned  vs.  manned  vs.  multimanned  systems,  etc. 
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Two  respondents  said  that  the  material  was  too  generalized  and  should  be  more 
specific.  One  said:  Surely  this  is  not  purported  to  be  a  complete  list  of  design  concepts 
which  impact  maintainability  and  operability? 

Two  respondents  reiterated  questions  concerning  the  data  base:  It  would  be  good 
to  get  a  bigger  data  base  of  opinions.  We  are  forced  to  assume  that  this  work  is  valid;  the 
information  in  the  technical  report  covering  the  development  of  the  guide  would  be 
crucial  to  determining  if  the  data  really  are  valid. 

Comment  on  specific  topics  in  Section  1  included:  Expand  definition  of  the 
term  "technical  fea  bility"  and  add  an  explanation  of  "technical  and  practical  risks."  The 
calibration  of  the  asurement  scale  (e.g.,  plus  or  minus  30  percent)  is  not  explained 
satisfactorily.  On<  >uld  review  standard  hardware  versus  system  complexity  and  mean 
time  to  repair. 

Two  comments  suggested  that  the  data  were  presented  in  something  of  a 
vacuum:  The  effect  of  combinations  of  design  concepts  should  be  considered;  this  is 
usually  the  case  in  actual  maintenance  design  concepts.  The  technical  aspects  of  the 
system  should  be  tied  in  to  this  discussion;  design  concepts  that  might  help  manpower 
should  be  considered  in  the  context  of  maintaining  the  required  level  of  technical 
performance  of  the  system. 

Two  comments  suggested  that  the  information  was  not  complete  unless  it 
considered  several  additional  design  concepts;  these  design  concepts  are  presented  in  a 
later  section. 

5.  Understandable?  (10) 

One  commentator  said:  1  like  this  type  of  presentation  as  it  is  easy  to  spot 

trends. 

Four  comments  suggested  problems  with  the  data:  Formats  bothered  me  at  first. 
It  is  not  organized  in  a  useful  form.  It  took  a  few  minutes  to  figure  out  these  charts;  a 
specific  point  of  confusion  is  the  fact  that  the  "0"  on  the  charts  is  equivalent  to  the 
"baseline"  design  concept;  this  was  not  explained  or  I  missed  it.  The  charted  data  are  hard 
for  the  designer  to  translate  into  hardware  requirements. 

Two  comments  suggested  difficulty  in  understanding  Table  1  and  suggested  that 
a  good  explanation  is  needed. 

Figure  4  was  also  a  problem:  Figure  4  (and  other  initial  figures  in  each  section) 
should  be  set  apart  from  the  text  more  clearly,  perhaps  with  a  solid  line  across  the  page. 
This  would  also  highlight  the  purpose  of  the  figure.  I  like  the  use  of  cross-reference 
tables  in  Sections  1  and  2. 


One  respondent  stated:  Format  is  okay  but  the  content  is  misleading.  A  better 
approach  would  be  to  show  sigma  bars  around  responses. 

6.  Should  be  Changed?  (26) 


Several  comments  under  this  question  echoed  sentiments  and  suggestions  expres¬ 
sed  earlier  and  will  not  be  repeated  here.  A  large  number  of  comments  were  made  under 
this  section;  they  are  grouped  below  according  to  the  main  focus  of  each  comment. 


Three  commentators  did  not  like  the  basic  approach  and/or  the  content  of  the 
data  in  the  guide:  As  presented,  it  is  an  attempt  to  quantify  and  legitimatize  some 
subjective  data  and  is  not  useful.  The  generality  of  the  operational  design  concept  is  too 
great;  what  does  "operational  simplicity"  really  mean  to  the  rater?--minimizing  options  in 
an  inherently  complex  system  may  still  leave  the  operator  with  difficult  tasks  to  master. 
Curves  2  through  5  on  Pages  1-1  through  1-17  do  not  all  appear  to  be  logical  or  consistent. 

Several  commentators  indicated  that  the  material  should  be  reorganized  and 
reformatted:  Information  would  be  useful  if  it  included  a  good  index.  Information  should 
be  reviewed  and  reorganized.  Simplify  it  by  changing  and  improving  some  concepts  that 
are  obviously  not  applicable.  Should  be  revised  and  streamlined  to  include  only  those 
items  that  actually  affect  a  system  design  criterion.  Drop  profile  write-ups  from  the 
description  section  on  the  concepts;  the  discussion  of  the  profile  detracts  from  the 
presentation;  the  reader  can  get  the  same  information  by  examining  the  chart  himself.  I 
believe  it  would  be  more  convenient  if  the  graphs  were  presented  as  numerical  data  in  a 
single  table. 

Several  respondents  remarked  that  the  material,  as  presented  in  Section  1,  was 
inordinately  difficult  to  understand:  Text  material,  Pages  1-1  through  1-5,  is  very 
difficult  to  understand  without  rereading  it  several  times.  Eliminate  "taxonomy"  as  its 
meaning  is  not  clear;  use  normal  words!  Provide  more  complete  explanations.  Explain  the 
system  design  criteria  in  Section  1;  although  these  definitions  do  appear  in  Section  2,  they 
don't  help  the  reader  very  much  when  he  is  still  in  Section  1  and  needs  to  know  the 
information. 

A  couple  of  commentators  had  difficulty  finding  the  definition  of  "technical 
feasibility":  No  definition  of  "technical  feasibility"  was  found  in  the  publication  although 
it  is  presumably  a  measure  of  the  probability  that  the  system  design  concept  could  be 
implemented;  perhaps  a  brief  definition  of  the  term  and  how  it  was  derived  could  be 
included.  Show  technical  feasibility  in  Table  1. 

Several  respondents  expressed  a  desire  for  improvements  in  the  measurement 
scale:  The  percent  values  of  each  criterion  are  not  necessarily  comparable  units  (e.g., 
how  does  a  20  percent  increase  in  acquisition  costs  compare  to  a  10  percent  decrease  in 
MTBF?).  Presentation  technique  is  good  but  it  would  be  helpful  to  have  some  guides  on 
the  importance  of  10,  20,  30  percent  impact;  how  much  is  necessary  in  order  to  cause  a 
design  modification?  Need  a  way  of  showing  the  relationship  (weighting)  between  items. 

Two  commentators  focused  on  the  size  of  the  group  of  the  judges  and  problems 
with  consensus:  Try  to  get  mce  than  32  opinions  for  consensus.  The  data  base  should  be 
expanded  and  upgraded  with  fewer  indecision  marks. 

One  senior  program  manager  suggested:  Some  changes  need  to  be  made  in  this 
data  base.  The  way  "Technical  feasibility"  is  defined  in  the  guide  doesn't  tell  me 
anything.  The  profile  evaluation  could  be  more  specific;  at  this  point  it's  not  too  useful 
for  deciding  between  controversial  items.  I  don't  know  what  the  percentages  mean. 

Another  senior  program  manager  suggested:  A  methodology  should  be  developed 
to  relate  types  of  equipment  and  missions  to  specifications  for  the  appropriate  BITE 
penetration,  spares  planning,  etc.  How  about  including  a  designer's  consolidated  and 
comprehensive  checklist? 
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Table  1  shows  how  respondents  rated  the  understandability  and  utility  of  each  of  the 
21  design  concepts.  The  number  of  responses  upon  which  the  percentages  are  based  varied 
from  38  to  44,  depending  on  the  design  concept  and  the  type  of  question.  In  some  cases, 
respondents  markings  were  illegible;  in  others,  respondents  chose  not  to  mark  a  category 
for  a  number  of  reasons:  problems  in  understanding  the  design  concept,  disagreement 
with  a  specific  design  concept,  or  disagreement  with  the  general  approach  taken  in  the 
guide.  These  comments  were  included  in  the  comment  data  base.  The  "Utility" 
percentages  are  based  on  38  to  43  responses;  and  the  "Understandable"  percentages,  on  41 
to  44  responses.  Since  responses  indicated  a  high  level  of  understanding  of  all  21  design 
concepts,  the  following  paragraphs  will  focus  on  their  utility. 

1.  Equipment  Layout  to  Facilitate  Maintenance.  (3) 

Comments  included:  This  is  the  most  important  design  concept  in  my  opinion. 
This  design  concept  needs  categorizing  relative  to  systems  size— 1 -box,  10-box,  possible 
compartment  locations,  submarine  or  surface  ship?  This  design  concept  also  impacts 
MTBF  via  special  flex  cables,  smaller  partitioning  with  worse  cooling,  etc. 

2.  LRUs— No  Spares.  (21) 

Almost  half  of  the  respondents  indicated  that  this  design  concept  should  be 
dropped:  Drop  it;  this  is  impractical  from  a  realistic,  real  life  standpoint;  it  is  not  cost- 
effective.  This  philosophy  would  apply  only  to  a  small  number  of  systems  since  integrated 
circuits  are  so  prevalent  and  these  are  essentially  not  repairable.  Should  be  dropped 
because  of  its  adverse  effect  on  overall  operational  capability.  It  would  be  a  logistics 
nightmare  to  supply  each  ship  with  "piece  parts";  also,  the  distribution  and  inventory  costs 
would  be  significant.  Requires  an  extremely  knowledgeable  person  to  repair  the 
equipment.  Only  useful  in  nonessential  equipment.  There  is  no  guarantee  that  a  defective 
LRU  can  be  repaired— this  puts  the  equipment  completely  out  of  service.  No  design  work 
is  done  to  this  concept  any  more;  thus,  including  it  in  the  guide  is  a  waste.  This  is  just  a 
space  filler  for  the  guide;  there  is  a  33-percent  disagreement  on  its  impact,  and  there  are 
no  practical  differences  in  any  event.  I  can't  conceive  of  a  no-spares  situation  for  a 
primary  combat  system  unless  the  design  employs  a  great  deal  of  redundancy;  you  just 
can't  wait  for  LRU  repair  in  many  cases.  Too  much  risk  that  complete  capability  could  be 
lost  for  long  periods.  The  rate  of  technological  advance  is  such  that  the  use  of  this 
holdover  concept  would  produce  proportionately  greater  acquisition  costs  with  time. 
Components  would  have  to  be  spared  and  more  sophisticated  test  equipment  and  skills 
training  would  be  required.  Should  be  dropped  since  we  do  not  consider  personnel  to  be 
capable  of  onboard  repairs;  the  no-spares  concept  for  most  Navy  electronics  equipment  is 
never  considered.  For  a  complex  system  of  high  value,  this  is  not  a  viable  concept. 

Three  commentators  indicated  that  the  concept  should  be  modified.  One  said 
that  the  definition  of  LRU  was  wrong,  that  it  could  include  any  type  of  assembly.  One 
pointed  out  the  questionable  value  of  having  all  technicians  trained  to  handle  repair  of  PC 
boards.  The  third  commentator  suggested  that  Design  Concepts  2  through  5  should  be 
combined  in  some  fashion  to  achieve  a  better  logical  categorization;  he  said  that  Design 
Concepts  2,  3,  and  4  require  maintenance  at  some  level,  whereas  Design  Concept  5 
requires  no  additional  maintenance. 


Another  commentator  suggested  that  Design  Concepts  2  through  5  were  all 
viable  and  worth  keeping;  however,  he  presented  the  following  insight  on  problems  with 
sparing:  Decisions  regarding  LRU  spares  are  too  often  made  at  a  provisioning  conference 


Table  1 


Reactions  of  Commentators  to  Specific  Design  Concepts 


DESIGN  CONCEPT 

Understandable 
(%  Responding) 

I  Don't 

I  Under-  Under¬ 
stand  it  stand  it 

USEFUL: 

Keep 

it 

Utility 

(%  Responding) 

:  USEFUL:  USELESS: 
Modify  Drop 

it  it 

1.  Equipment  layout  to  facilitate  maintenance 

100 

0 

87 

8 

5 

2.  LRUs— No  spares 

100 

0 

42 

12 

46 

3.  LRUs— Spares  with  onboard  repair 

100 

0 

68 

20 

12 

4.  LRUs— Spares  with  remote  repair 

100 

0 

83 

10 

7 

5  LRUs— Spares  with  throwaway  maintenance 

100 

0 

71 

17 

12 

6.  "Overdesign"  for  reliability 

anc  maintenance 

100 

0 

65 

28 

7 

7.  Embedded  computers 

95 

5 

35 

51 

14 

8.  Automatic  performance  monitoring 

98 

2 

64 

31 

5 

9.  Built-in  test  equipment 

100 

0 

74 

21 

5 

10.  Buiit-in  troubleshooting  logic  aids 

98 

2 

69 

26 

5 

11.  Automatic  fault  localization 

98 

2 

65 

28 

7 

12.  Standard  hardware  components 

100 

0 

67 

21 

12 

13.  Standard  hardware— Cards/LRUs 

100 

0 

66 

19 

15 

14.  Standard  hardware— Functional  units 

100 

0 

55 

21 

24 

15.  Standard  hardware— Subsystems 

100 

0 

73 

17 

10 

16.  Operational  simplicity 

100 

0 

81 

12 

7 

17.  Built-in  operator  performance  aids 

100 

0 

78 

10 

12 

18.  Automatic  decision  making 

100 

0 

62 

30 

S 

19.  Automatic  information  transmit 

ana  display 

98 

2 

71 

19 

10 

20.  Built-in  training  capability 

100 

0 

72 

15 

13 

21.  Combined  opera tor/maintainer  functions 

100 

0 

61 

18 

21 

Note.  31  percent  of  the  respondents  indicated  that  additional  design  concepts  should  be  included. 


JO 


and  are  made  on  the  basis  of  funds  that  the  Navy  has  at  a  particular  point  in  time  rather 
than  on  the  basis  of  the  designer's  concepts.  As  a  result,  the  system  is  supported  in 
accordance  with  a  later  logistic  concept  rather  than  the  original  design  concept.  This 
results  in  a  mismatch  between  sparing  support  and  system  design.  To  correct  this,  a  firm 
decision  should  be  made  at  the  time  that  the  specification  is  issued  on  what  the  LRU 
sparing  philosophy  will  be  as  well  as  the  point-of-repair. 

3.  LRUs— Spares  With  Onboard  Repair.  (7) 

Most  of  the  seven  comments  suggested  that  this  design  concept  be  dropped  or 
modified.  Those  who  said  it  should  be  dropped  stated:  This  would  keep  the  system  off  the 
air  during  maintenance.  This  has  all  the  disadvantages  of  Design  Concept  No.  2,  with  the 
added  cost  of  a  set  of  spares.  Since  most  electronics  is  now  mounted  on  printed  circuit 
boards,  which  require  a  high  degree  of  skill  to  repair,  this  option  was  never  considered 
other  than  in  an  emergency;  I  consider  any  shipboard  repair  to  be  very  detrimental  to 
system  reliability  later  on. 

Those  who  said  that  the  concept  should  be  modified  stated:  In  many  cases 
onboard  repair  will  not  be  cost-effective,  since  it  is  likely  that  more  test  equipment  and 
training  will  be  required.  Limit  the  types  of  onboard  repair.  Minimize  onboard  repair. 

One  commentator  introduced  a  wider  perspective  in  considering  this  design 
concept,  along  with  Design  Concepts  4  and  5:  How  about  adding  the  concept  of  onboard- 
repair/remote-repair/throwaway  philosophy  based  on  proper  logistic  support  analysis  and 
life  cycle  cost  considerations?  This  is  current  reality. 

4.  LRUs— Spares  With  Remote  Repair.  (5) 

One  commentator  said  to  drop  this  concept:  This  system  has  the  same 
limitations  as  Design  Concepts  2  and  3  but  utilizes  an  unworkable  supply  concept. 

Two  respondents  said  that  the  concept  should  be  modified:  A  compromise 
between  Design  Concepts  2  and  4  could  be  worked  out;  however,  I  am  not  sure  how  you 
would  select  which  spares  to  carry.  In  Design  Concept  4,  the  concept  needs  a  standard 
Navy  circuit  board  size  in  order  for  the  concept  to  work;  a  combination  of  Design 
Concepts  4  and  5  would  work  even  better  where  you  would  have  throwaway  for  a  given 
dollar-value  board  and  remote  repair  for  the  remainder. 

Two  commentators  endorsed  this  concept:  With  proper  planning  and  support,  this 
concept  can  result  in  adequate  support  at  moderate  costs,  either  with  depot  or  contractor 
repair.  Although  this  concept  has  merit,  I  feel  that  "remote  repairs"  simply  means  "at  a 
contractor  facility." 

5.  LRUs— Spares  With  Throwaway  Maintenance.  (6) 

Three  of  the  six  comments  indicated  that  this  design  concept  should  be  dropped: 
It  has  the  same  limitations  as  Design  Concepts  2,  3,  and  4.  It's  possibly  workable,  but  at  a 
high  price,  probably  too  high  in  view  of  the  trend  to  pack  a  lot  of  hardware/software  onto 
a  single  board.  It  costs  too  much. 

Three  comments  suggested  that  the  concept  be  modified:  No  system  can  afford 
complete  spares  or  throwaway  maintenance;  this  should  be  modified  to  read  "partially 
spares  with  remote  repair  and  partially  spares  with  throwaway  maintenance."  Throw¬ 
aways  should  be  determined  by  cost.  In  deciding  on  a  throwaway,  one  must  also  consider 
the  cost  of  repair  (e.g.,  paperwork,  test,  evaluation,  actual  repair,  etc). 
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>.  "Overdesign"  for  Reliability  and  Maintenance.  (10) 


Two  comments  noted  the  current  status  of  this  concept:  Some  current  system 
specifications  already  require  that  you  consider  overdesign.  Overdesign  has  proven  useful 
on  submarines,  especially  stable  circuits.  High  reliability  JAN  parts  and  components  are 
already  required  in  the  general  and  equipment  specifications;  this  concept  would  be  valid 
for  redundant  circuits  and  backup  mode  equipment. 

Several  comments  reflected  different  emphases  in  discussions  of  this  concept: 
Redundancy  is  mentioned  but  1  suggest  that  a  little  more  emphasis  be  given  to  redundancy 
in  circuit  design.  In  the  explanation,  I  would  emphasize  redundancy  much  more  than 
component  ratings;  we  are  now  required  to  use  very  high-rated  components,  but  failure  is 
more  likely  to  occur  in  the  board,  solder  joints,  connectors,  etc.;  so  redundancy  is  the 
main  design  tool.  I  believe  we  consider  hydraulic/mechanical  interlocks  in  lieu  of 
additional  electronics.  Provide  an  example  of  the  use  of  each  design  concept  and,  if 
several  methods  are  used  to  achieve  a  particular  design  concept,  provide  as  many 
examples  as  methods  (e.g.,  higher-rated  components,  system  redundancy,  ultrareliable 
components). 

Three  comments  focused  on  possible  difficulties  with  this  design  concept:  If  a 
circuit  seldom  fails,  the  sailors  will  have  little  practice  in  repairing  it;  they  will  have 
forgotten  their  training  and  must  be  very  competent  to  fix  it.  Difficult  to  implement 
effectively.  The  timely  availability  of  higher  quality  parts  is  a  real  problem  in 
implementing  this  design  concept;  in  considering  this  concept,  you  should  include  the 
effects  of  likely  scheduling  difficulties. 

7.  Embedded  Computers.  (25) 

The  large  number  of  commentators  on  this  topic  almost  universally  criticized  the 
guide's  treatment  of  this  design  concept.  One  commentator  stated:  Your  concept  is 
naive;  you  have  erroneously  made  the  baseline  to  be  1950  technology.  Page  1-21  indicates 
a  definite  "cultural  lag"  in  accepting  microprocessor  technology.  Don't  confuse  micro¬ 
processors,  which  are  part  of  a  technology,  with  "distributed  processing,"  which  may 
become  a  system  requirement  in  complex  multifunction,  multicomputer  systems.  Also, 
embedded  computers  seem  to  be  out  of  place  with  the  other  design  concepts.  There  are 
other  more  system-relevant  computer  configurations  decisions  that  impact  maintenance 
and  maintenance  philosophy;  these  include  redundancy  and  reconfiguration. 

Other  commentators  found  fault  with  the  presentation  on  embedded  computers 
in  the  guide:  This  is  not  the  same  type  of  concept  as  the  others;  maintenance  people  can 
learn  about  troubleshooting  embedded  computers  as  easily  as  any  other  logic  circuit.  I 
question  whether  the  new  microprocessors  are  as  unreliable  as  the  profile  indicates. 
Embedded  computers  can  be  treated  like  any  other  LRU  at  a  remote  repair  depot;  they 
can  be  tested  as  a  total  "black  box"  with  a  sophisticated  card  analyzer.  The  description  of 
embedded  computers  does  not  have  to  be  true;  the  data  shows  a  great  misunderstanding  of 
the  technique.  Consideration  of  embedded  computers  as  LRUs  with  remote  or  throwaway 
maintenance  would  alleviate  the  shipboard  maintenance  requirements  for  such  units.  I 
disagree  with  the  definition  as  stated  and  the  group  that  provided  the  judgmental  analysis; 
this  is  an  extremely  immature  technical  attitude;  I  could  see  constraints  on  the  use  of 
microprocessors,  but  not  as  stated  in  the  guide.  The  degree  of  difficulty  in  maintaining 
embedded  computers  is  dependent  on  the  maintenance  philosophy  that  is  employed. 
Embedded  computers  can  be  readily  checked  by  automatic  test  equipment,  which  nullifies 
the  adverse  impacts  under  System  Design  Criteria  A,  B,  C,  E,  and  F. 
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A  large  number  of  comments  focused  on  the  actual  benefits  of  embedded 
computers:  Embedded  computers  are  useful  for  implementing  Design  Concepts  8,  9,  and 
10  and  may  be  cost-saving.  Embedded  computers  may  lead  to  better  fault  isolation.  With 
adequate  BITE,  embedded  computers  can  be  effective  in  permitting  maintenance  by  lower 
skill  levels.  Consideration  should  be  given  to  the  diagnostic  capability  embedded 
computers  can  provide.  Totally  disagree;  I  think  most  people  miss  the  point;  embedded 
processors  do  not  require  standard  computer-type  maintenance  but  should  be  treated  as 
any  other  logic  card  with  some  built-in  test  capability;  embedded  processors  are,  in  fact, 
the  best  means  of  accomplishing  reduced  manpower  for  complex  systems.  If  embedded 
computers  were  combined  with  automatic  performance  monitoring  and  built-in  trouble¬ 
shooting  aids,  then  the  negatives  on  the  data  charts  should  become  positives.  Disagree-- 
embedded  computers  and  implanted  test  points  provide  fault  detection/isolation 
capabilities;  this  evaluation  is  completely  in  error.  Suitable  application  of  limited 
embedded  processing  can  save  centralized  processors,  which  create  their  own  problems  of 
programs  and  control.  If  properly  implemented,  embedded  computers  can  enhance 
maintenance. 

Three  comments  focused  on  the  widespread  and  growing  use  of  embedded 
computers:  It  would  be  hard  to  stop  or  even  slow  down  the  use  of  microprocessors;  very 
large-scale  integration  is  the  "wave  of  the  future."  This  concept  will  get  greater 
emphasis  as  large-scale  integration  and  very-large-scale  integration  matures  and  the 
military  better  learn  to  handle  proper  maintenance  training.  I  believe  this  is  a  more 
common  practice  than  the  text  implies;  the  descriptive  text  seems  obviously  biased 
against  the  concept.  The  profile  is  accurate  for  now,  but  by  the  time  this  is  published,  it 
will  have  drifted  to  the  right  about  10  percent;  this  is  the  most  dynamic  design  concept 
right  now,  and  the  results  in  each  case  are  highly  dependent  on  the  quality  and  type  of 
implementation. 

8.  Automatic  Performance  Monitoring.  (5) 

The  five  comments  on  this  topic  were  quite  varied.  They  included:  Automatic 
performance  monitoring  is  okay  if  it  is  coupled  with  attempts  to  automatically  restore 
degradation.  You  should  include  PMFL  (Performance  Monitoring  and  Fault  Localization); 
this  would  increase  initial  design  costs  but  greatly  reduce  training,  MTTR,  and  required 
experience  for  personnel.  Do  you  mean  performance  or  operational  readiness 
monitoring?- -true  performance  monitoring  by  automatic  means  would  be  far  more 
adverse  than  shown.  Because  the  chart  shows  that  33  percent  of  the  system  design 
concepts  were  marked  "Substantial  disagreement  on  impact,"  this  concept  should  be 
modified  or  possibly  deleted.  One  commentator  offered  the  following  observations  about 
automated  performance  monitoring:  The  impact  on  MTBF  will  not  be  adverse  if  the 
interfaces  are  fail-safe. 

One  commentator  believed  that  Design  Concepts  8,  9,  10,  and  11  should  be 
combined  into  one  or  two  design  approaches. 

9.  Built-in  Test  Equipment.  (2) 

One  commentator  said:  This  design  concept  has  the  greatest  potential  for 
reducing  maintenance  manpower;  the  "high  initial  acquisition  cost"  is  really  a  good 
investment. 

The  second  commentator  said:  I  don't  believe  the  chart  reflects  a  true  picture 
for  the  impact  on  System  Design  Criteria  D,  G,  and  N. 
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10.  Built-in  Troubleshooting  Logic  Aids,  (4) 

One  commentator  stated  that  the  design  concept  needed  a  better  definition  and 
specific  examples.  A  second  said  that  this  concept  should  be  used  in  conjunction  with 
separate  test  equipment  and  documentation  as  a  transition  strategy  between  the  baseline 
and  fully  built-in  troubleshooting  logic  aids. 

One  respondent  stated:  Too  much  sophistication  in  troubleshooting  aids  can 
drive  the  maintainer  away  from  getting  a  "feel"  for  the  equipment.  The  maintainer  must 
have  a  functional  understanding  of  the  equipment. 

Two  commentators  were  concerned  about  software  costs  for  implementing  this 
design  concept:  Software  costs  are  underestimated.  This  design  concept  has  a  great 
potential  for  reducing  maintenance  manpower,  but  the  software  aspects  could  drive  costs 
up. 


11.  Automatic  Fault  Localization.  (8) 

Two  commentators  indicated  difficulty  with  the  definition  of  this  design 
concept.  One  commentator  wanted  it  expanded  and  examples  provided.  The  other  stated 
that  the  definition  of  automatic  fault  localization  to  three  printed  circuit  cards  is 
unrealistic,  that  such  a  low  number  of  cards  would  drive  the  costs,  the  computer 
programming,  and  the  size  up  too  far.  A  third  commentator  said  that  if  the  amount  of 
circuitry  required  to  provide  automatic  fault  localization  is  substantial,  the  maintenance 
of  such  circuitry  must  be  considered. 

Several  respondents  saw  the  virtues  in  automatic  fault  localization:  This 
concept  will  grow  with  the  use  of  very-large-scale  integration;  it  has  great  long-range 
potential  for  manpower  savings.  Fault  localization  pays  for  itself,  particularly  in  MTTR; 
however,  the  tendency  is  to  go  overboard  with  fault  localization— better  to  concentrate  on 
the  most  common  failures. 

Two  commentators  disagreed  concerning  the  negative  impact  on  MTBF.  One 
stated  that:  MTBF  is  not  negative  because  MTTR  is  the  overriding  factor. 

Another  commentator  indicated  that  MTBF  would  not  be  affected  negatively,  as 
shown  in  the  charts,  especially  if  fail-safe  interfaces  existed  between  the  test  system  and 
tactical  functions.  However,  he  cautioned,  while  automatic  fault  localization  does 
significantly  improve  a  portion  of  the  MTTR,  it  won’t  do  it  to  the  extent  shown  in  the 
chart  if  you  take  into  account  all  the  steps  and  procedures  involved  in  mean  logistics 
downtime  (MLDT). 

12.  Standard  Hardware  Components.  (7) 

One  commentator  heartily  endorsed  this  design  concept:  Your  improvement 
percentages  for  System  Design  Criteria  A,  B,  C,  E,  F,  H,  and  D  are  conservative.  Printed 
card  board  design  may  vary  across  contractors,  but  other  system  specification  require¬ 
ments  can  make  further  improvements  through  standardization— standardized  cabinets, 
standard  printed  card  board  nests,  access,  replace  without  adjustment,  etc. 

One  senior  member  of  the  design  community  said:  If  the  desired  system 
capabilities  are  to  be  met,  the  Navy  supply  system  should  be  improved  to  use  this  concept. 
Alternatively,  the  Navy  could  help  with  vendor  component  availability.  The  utility  of  the 
concept  depends  on  how  it  is  managed. 
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Another  commentator  pointed  out  a  practical  problem  in  implementing  this 
concept:  Somewhere  or  somehow,  a  designer  must  be  made  aware  of  where  data  on 
standard  Navy  hardware  is  available  for  his  use.  One  commentator  stated  that  standard 
hardware  should  be  used  where  practicable  but  not  across  the  board. 

T wo  commentators  rejected  use  of  this  concept:  Since  I  see  very  little  repair  at 
the  component  level  aboard  ship,  standardizing  at  the  transistor,  resistor,  etc.,  level 
would  not  accomplish  much;  I  would  drop  this.  The  designer  cannot  keep  up  with 
technology  and  new  requirements  if  he  uses  this  concept. 

13.  Standard  Hardware-Cards/LRUs.  (10) 

One  comment  supported  this  design  concept:  Very  effective  in  reducing 
acquisition  cost  and  improving  MTBF  via  higher  production.  Four  comments  indicated 
rejection  of  the  concept:  Not  compatible  with  current  techniques;  this  is  really  1960's 
stuff  that  did  not  work.  Strongly  disagree  that  this  is  a  useful  concept.  Standard  cards 
tie  you  to  obsolete  technology;  since  the  development  cycle  is  very  long,  very  few  people 
would  want  to  start  the  cycle  with  old  technology;  also,  this  design  concept  would  produce 
a  very  costly  system  of  low  reliability  (too  many  parts).  Today's  systems  are  so  complex 
and  the  goals  are  so  precisely  defined  that  such  standards  would  be  awkward  to  use.  For 
example,  the  digital  function  cards,  which  were  general  purpose,  that  were  offered  by 
numerous  companies  in  the  1960's  are  practically  nonexistent  today. 

Other  problems  in  using  this  design  concept  were  mentioned:  This  is  generally 
specified  in  the  individual  equipment  specification  and  is  not  optional  to  the  designer. 
This  concept  is  feasible  but  it  is  limited  by  performance  requirement. 

One  respondent  expressed  disbelief  that  standard  cards  exist. 

14.  Standard  Hardware-Functional  Units.  (10) 

Three  commentators  believed  that  this  concept  should  be  included  with  No.  1 3. 
One  of  these  suggested  it  should  also  be  combined  with  No.  15. 

Several  commentators  saw  problems  in  implementing  this  concept:  Not 
practical,  every  system  has  its  own  requirements.  Difficult  to  accomplish.  Most 
hardware  systems  have  unique  requirements;  unless  standard  units  were  very  cheap,  this 
would  be  a  very  expensive  way  to  design.  Complex  functional  units  are  rarely 
exchangeable  between  systems.  With  the  advent  of  very-large-scale  integration,  func¬ 
tional-unit  packaging  for  large  signal  and  data  processing  systems  becomes  less  tech¬ 
nically  feasible  and  would  increase  manufacturing  costs;  production  technology  for  very- 
large-scale  integration  and  discrete  components  to  be  put  on  the  same  board  may  be 
difficult. 


15.  Standard  Hardware-Subsystems.  (6) 

One  comment  suggested  that  the  beneficial-impact  percentages  for  System 
Design  Criteria  A,  B,  C,  and  F  were  conservative. 

Three  respondents  related  this  design  concept  to  computer  systems:  Think  this 
only  applies  to  the  computer  world;  it  doesn't  mean  much  to  other  designers.  I  am  not 
aware  of  any  standard  hardware  subsystems  unless  this  is  addressing  items  such  as  the 
UYK-20,  UYA-4,  etc.;  if  this  is  the  case,  it  should  be  made  clear.  This  design  concept 
must  be  modified  to  reflect  that  the  approach  results  in  the  use  of  outdated,  inappropriate 
hardware  (e.g.,  UYK-7  and  UYK-7  watercooled  computers). 


25 


Two  comments  suggested  problems  in  implementing  this  design  concept: 
Standard  hardware  subsystems  cannot  provide  acquisition-cost  advantages  unless  the 
equipment  and  documentation  are  available  early  in  the  design  process;  also,  the  standard 
subsystems  cannot  be  too  elaborate  (i.e.,  provide  everything  for  everybody).  Standard 
hardware  at  the  subsystem  level  is  impractical  because  it  requires  too  much  designer 
research  to  implement. 

16.  Operational  Simplicity.  (5) 

Two  of  the  five  comments  indicated  problems  in  the  definition  of  this  design 
concept:  Need  better  definition  of  simplicity  (e.g.,  does  it  mean  many  simple  modes  are 
easier  than  fewer  complex  modes?).  Also,  need  guidance  on  MMI  criteria.  The  choices 
for  operational  simplicity  need  to  be  made  more  specific;  also,  you  must  consider  the  cost 
of  this  simplicity  in  terms  of  a  reduction  of  total  mission  performance  capability. 

Two  comments  indicated  the  problems  in  implementing  this  concept:  The 
problem  is  that  no  two  areas  of  discipline  agree  on  what  controls  they  need  and  what  they 
can  do  without.  Where  electronics  is  used  to  control  complex  mechanical  equipment,  like 
missile  launchers  and  gun  mounts,  the  mechanical  maintenance  requirements  limit  the 
degree  of  operational  simplification. 

17.  Built-in  Operator  Performance  Aids.  (2) 

Of  the  two  comments  made  under  this  topic,  one  could  not  be  interpreted.  The 
other  stated:  Maybe  there  should  be  two  categories  of  design  concepts;  one  would  be 
General  Concepts  and  the  other  Special  Concepts.  Design  Concept  17  would  be  in  the 
Special  category.  It  would  only  be  of  value  in  a  system  where  the  operating  sequences 
were  so  weird  that  cueing  was  necessary. 

18.  Automatic  Decision-Making.  (6) 

Three  comments  indicated  a  need  to  clarify  the  definition  of  the  concept:  Needs 
clarification  about  the  nature  and  level  of  decisions  to  be  made.  Complicates  matters  and 
can  be  misleading.  Does  operational  capability  and  effectiveness  diminish  when  the 
automatic  decision  is  unacceptable?  Needs  to  deal  with  the  case  where  an  operator  may 
override  the  automatic  decision. 

One  commentator  stated  that  the  design  concept  may  reduce  required  operator 
skills  but  greatly  increase  maintainer  skills. 

One  commentator  offered  the  following  to  explain  the  trend  toward  automatic 
decision-making:  Tactical  threats,  reaction  time  requirements,  complexity  of  processing, 
multiple  target  AAW  handling,  etc.  are  driving  some  systems  in  this  direction.  The  slower 
moving  scenarios  of  ASW  can  be  approached  differently,  as  can  surface  warfare  in  a 
limited  number  of  cases. 

19.  Automatic  Information  Transmit  and  Display.  (5) 

Two  comments  indicated  problems  in  implementing  this  design  concept  in  certain 
cases:  You  can't  do  this  with  individual  systems;  it  is  only  possible  when  dealing  with  a 
whole  ship.  Requires  total  integration  into  ship  routine. 

Two  commentators  said  that  there  was  no  alternative  to  using  the  design 
concept:  This  is  basic  to  most  modern  systems;  who  would  design  without  it?  In  my 
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opinion,  today's  technology  has  made  the  alternative  trivial,  particularly  when  trying  to 
conserve  personnel. 


One  commentator  offered  the  following  to  explain  why  alternatives  to  the  design 
concept  may  still  be  viable  for  Navy  systems:  In  the  case  of  this  design  concept, 
technology  is  competing  with  traditional  Navy  schools  of  thought  concerning  "ship  driving" 
and  "ship  fighting."  Improvements  may  be  accepted  in  selected  pieces  of  equipment,  but 
the  OOD  will  not  let  a  technically  bright  contractor  take  over  the  entire  operation  of  his 
bridge.  In  casualty  conditions,  when  computer-aided  displays  have  failed,  the  runners, 
talkers,  etc.,  may  still  be  used.  In  considering  the  limiting  cases  of  survival,  safety  of  the 
ship,  navigation  with  casualty  damage,  responsible  officers  will  revert  to  basic,  no-frills 
information  communication;  a  contractor  will  not  be  allowed  to  "design  it  out  of  the 
system." 


20.  Built-in  Trainer  Capability.  (6) 

One  of  the  comments  could  not  be  interpreted.  Three  comments  expressed 
disagreement:  This  area  is  such  a  plus  as  to  be  almost  mandatory;  I  do  not  agree  with  the 
.76  for  technical  feasibility— it  should  be  more  like  .96.  I  don't  believe  that  an  adverse 
impact  of  -10  percent  is  true  for  System  Design  Criterion  A;  training  equipment  will  be 
common  to  system  equipment,  certainly  no  more  complex  if  the  human  factors  engineer 
does  a  good  job.  Hardware  costs  are  underestimated. 

Two  respondents  suggested  that  the  design  concept  was  unnecessary:  03T 
onboard  ship  should  be  done  with  actual  operator  consoles;  there  is  no  need  to  make  new 
hardware.  It  is  not  utilized  enough  in  the  operational  environment  to  warrant  costs. 

21.  Combined  Operator/Maintainer  Functions.  (6) 

One  comment  endorsed  the  combined  function:  You  must  know  how  to  operate  it 
if  you  maintain  it;  have  a  common  person. 

Two  comments  criticized  the  combined  function:  The  complexity  of  modern 
systems  requires  full-time  maintenance  personnel  to  maintain  a  system.  Combining  the 
functions  would  require  a  highly-trained  technician  to  stand  long  watches  when  he  could 
be  gainfully  employed  in  maintenance. 

One  comment  showed  disagreement  with  one  of  the  statements  in  the  guide:  I 
don't  agree  that  "commonly"  two  separate  sets  of  personnel  operate  and  maintain. 

One  commentator  said:  To  my  knowledge,  gunners'  mates  must  operate  and 
maintain  their  own  equipment  and,  therefore,  there  is  no  tradeoff  here  for  our  equipment 
design  (launchers  and  gun  mounts). 

One  commentator  saw  the  following  trend:  The  1980s  will  see  a  significant 
departure  from  the  baseline.  Smaller  ships,  like  the  FFG  and  DDG,  will  invest  in 
combined  operator/maintainer  rates  and  training  to  minimize  crew  size. 

Suggested  Additional  Design  Concepts. 

Seventeen  respondents  made  comments  regarding  additional  design  concepts  to  be 
covered  in  the  guide. 

Two  suggestions  concerned  training  concepts:  How  about  shipboard  self-paced 
maintenance  training  as  a  concept  directed  toward  upgrading  technical  capability?  I 
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suggest  cross-training  of  maintainers  and  operators  on  different  types  of  systems  as  a 
design  concept. 

One  commentator  suggested  adding  the  complement  to  Design  Concept  1:  Design 
Concept  1,  design  for  maintenance,  is  half  of  human  engineering;  design  for  operability  is 
the  other  half.  Design  for  operability  should  be  added  to  the  list  of  design  concepts  to 
round  out  human  engineering. 

Three  comments  focused  on  functional  partitioning  as  a  design  concept:  Functional 
partitioning  to  one  LRU,  or  a  few  LRUs,  could  greatly  benefit  MTTR.  A  system 
partitioned  into  "small  black-box"  functions  would  be  a  good  design  approach;  each  box 
would  be  an  LRU  (e.g.,  power  supply,  logic  module,  A/D  or  D/A  converters,  display, 
keyboard,  etc.). 

One  commentator  seemed  to  combine  the  concept  of  functional  partitioning  with 
standardization:  Standard  modules  (PC  cards)  and  other  standard  entities  (cabinets,  power 
supplies,  etc.)  are  used  in  AEGIS.  Perhaps  this  concept  would  be  useful  whenever  a  new 
system  is  designed. 

One  commentator  suggested  standardization  for  operator  displays:  Operator  presen¬ 
tations  or  displays  should  be  standardized  (e.g,  signal-processing  displays  like  GRAM, 
WATERFALL,  PPI,  etc.). 

One  commentator  suggested  standardization  for  software:  Operation  and 

maintenance  would  be  impacted  by  standard  software  modules,  software  debug  aids, 
automatic  documentation,  etc. 

One  commentator  addressed  the  problems  concerning  Design  Concept  7:  Design 
Concept  7,  the  embedded  computer  or  microprocessor,  is  a  fact  of  life.  1  think  some 
additional  design  concepts  should  be  developed  which  make  the  embedded  computer 
acceptable.  For  example,  one  design  concept  could  utilize  a  microprocessor  where  the 
complete  unit  is  an  LRU  so  that  no  troubleshooting  is  required  within  the  microprocessor. 
A  second  concept  would  be  to  specify  that  maintenance  by  a  computer  specialist  is 
required  in  certain  cases  where  you  can  anticipate  that  the  traditional  rating  assigned  to 
the  equipment  is  not  qualified.  A  third  concept  would  be  use  of  redundant  embedded 
processors. 

Three  commentators  alluded  to  redundant  processing,  as  well  as  other  new  trends  in 
hardware/software  design:  Consider  multiple  redundancy  systems  with  majority  voting 
and/or  self-healing.  You  should  design  for  redundancy,  channelization,  and  graceful 
degradation  to  minimize  the  impact  of  failure;  also,  better  maintenance  and  training 
documentation  should  be  functionally  oriented.  Consider  distributed  computation  similar 
to  Design  Concept  7  but  with  greater  emphasis  on  reducing  mainframe  CPU;  this  will 
reduce  long-range  costs. 

Two  commentators  focused  on  concepts  to  reduce  maintenance  onboard  ship  through 
a  change  in  repair  philosophy:  Deferred  maintenance  is  a  major  design  concept;  as  an 
example,  in  the  DD  %3,  25  percent  of  organizational  maintenance  will  be  deferred  to  in¬ 
port  periods  where  it  is  done  by  roving  teams.  A  concept  is  zero  maintenance  (from  the 
crew's  viewpoint)  control  systems  with  sufficient  redundancy  that  they  operate  for  one  or 
more  years;  at  the  end  of  this  period,  a  team  of  experts  comes  aboard,  tests  and  stores 
full  redundancy,  and  buttons  up  the  system  for  at  least  one  more  year. 
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One  commentator  suggested  the  following  maintenance  concept:  Another  concept, 
not  liked  by  the  Navy  but  practical  in  the  fleet,  is  cannibalism;  controlled  cannibalism 
should  be  evaluated  as  a  design  concept. 

Another  commentator  suggested:  Shipboard  layout  to  facilitate  maintenance  would 
be  a  valuable  design  concept.  This  would  include  centralization  of  control,  spares  access, 
information  storage  and  retrieval,  tools  and  test  equipment,  readiness  assessment,  etc. 

Two  commentators  suggested  the  following  approaches  to  troubleshooting:  Use  of 
interactive  computer-aided  troubleshooting  and  maintenance,  using  an  off-line  computer. 
Use  of  fault  trees,  visual  fault  indicators,  and  discrete  LEDs  on  printed  circuit  boards. 

One  commentator  suggested  the  design  concept  of  minimizing  power  and  cooling 
requirements  in  system  design. 

General  Comments  on  Design  Concepts 

Several  comments  could  only  be  categorized  as  generally  applicable  to  the  entire 
discussion  of  design  concepts. 

One  commentator  stated:  All  these  concepts  have  some  value,  but  must  be  applied 
with  discretion.  The  designer  should  give  tradeoff  consideration  in  each  case. 

For  new  system  development  (1970-1980),  Design  Concepts  3,  4,  5,  6,  8,  9,  11,  13,  15, 
17,  18,  20,  and  21  will  most  likely  be  addressed  as  an  integrated  system  approach. 
Evaluating  them  as  they  would  interact  together  would  be  different  from  evaluating  them 
as  individual  concepts. 

One  commentator  said:  Any  systems  designer  with  any  degree  of  experience  is  aware 
of  all  of  the  concepts  listed  in  the  guide.  There  are  no  blacks  and  whites.  Different 
aspects  of  the  problem  require  different  tradeoff  considerations;  any  new  system  is  made 
up  of  standard  and  nonstandard  parts,  LRUs  which  are  repairable  onboard,  LRUs  which  are 
repairable  at  the  intermediate  and  depot  levels,  built-in  test  equipment,  etc. 

Another  commentator  echoed  the  admonition  to  make  design  decisions  tailored  to 
specific  cases:  The  more  automatic  aids  that  you  add  to  gear,  the  less  system  reliability 
you  can  expect.  Very  careful  tradeoffs  are  required  in  each  specific  case. 

One  evaluator  said:  Design  Concepts  16,  17,  19,  20,  and  21,  along  with  Concepts  8,  9, 
10,  and  11,  are  the  wave  of  the  future.  Design  Concept  18  will  be  overtaken  by  embedded 
computers  and  distributed  computation. 

Comments  from  Other  Sources 


Specific  Design  Concepts.  The  majority  of  comments  from  other  sources  echoed 
thoughts  and  suggestions  that  were  evident  in  those  made  by  respondents  to  Evaluation 
Packages  C  and  G.  To  repeat  these  comments  would  be  to  add  a  great  deal  of  redundancy 
to  this  section.  We  will  only  present  comments  that  provide  new  perspective  and  insights. 
The  following  paragraphs  contain  the  comments  without  introductory  remarks  and  are 
ordered  by  the  order  of  the  design  concepts;  the  last  few  paragraphs  contain  general 
comments  pertinent  to  all  or  most  of  the  design  concepts. 

1.  Design  Concept  1  lumps  together  several  design  approaches  that  should  be 
separated  because  different  systems  are  more  or  less  amenable  to  them.  These 


29 


approaches  include  modularization  by  function,  easy  access  for  maintenance, 
extensive/easily  understood  coding,  and  test  point  accessibility/design. 


2.  Design  Concepts  2  through  5,  on  LRUs,  should  be  combined.  They  should  treat 
the  general  problem  of  accessibility  of  spare  parts  via  intrasystem  communality,  built-in 
spares,  spare  kits,  lifetime  spares  purchased  at  time  of  acquisition,  etc.  Accessibility  of 
spares  is  an  important  element  that  can  facilitate  "Easter-egging"  and  minimized 
downtime.  (Easter-egging  denotes  the  practice  of  sequentially  removing  parts  of  a  system 
and  replacing  each  one  with  a  known  good  spare  until  a  system  begins  working.)  Accessible 
spares  is  the  key  in  Design  Concepts  2  through  5,  not  standardization  per  se.  Standardiza¬ 
tion  has  little  effect  on  manpower  utilization  even  though  it  may  greatly  affect  system 
downtime.  Also,  modularization  has  no  real  impact  unless  combined  with  accessibility /re- 
placeability  concepts  or  with  modularity  by  function  concepts. 

The  judges  have  failed  in  their  assessment  of  Design  Concepts  2  through  5 
concerning  LRUs.  They  have  not  fully  taken  into  account  the  full  inventory  and 
distribution  costs,  nor  the  talent  level  of  technicians  required,  when  onboard  repair  is 
considered.  The  cost  of  training  people  and  providing  facilities  for  organizational  level 
repair  is  inconceivable.  Throwaway  is  much  more  cost-effective.  As  an  example  of  how 
difficult  it  is  to  repair  complicated  boards,  IBM  has  had  trouble  repairing  some  of  their 
own  UYF-1  cards;  if  IBM  can  barely  do  it,  the  Navy  can't  possibly  do  it. 

3.  Design  Concept  6  misses  the  point;  overdesign  to  achieve  high  reliability  and 
maintainability  is  not  an  issue  in  manpower  savings  per  se.  "High  inherent  availability"  is 
a  proper  design  concept.  This  concept  allows  reliability/maintainability  tradeoffs  within 
the  perspectives  of  operational  constraints. 

4.  The  view  on  Design  Concept  7,  Embedded  Computers,  is  wrong.  An  across-the- 
board  statement  that  microprocessors  have  a  negative  impact  is  akin  to  someone  20  years 
ago  saying  we  should  not  use  transistors.  Maybe  there  should  be  a  limitation  on  the  use  of 
microprocessors  in  systems  that  are  maintained  by  unsophisticated  personnel,  but  do  not 
make  blatant,  blanket  statements  about  them.  Microprocessors  are  not  always  associated 
with  embedded  computers,  the  latter  being  16-,  or  32-bit  machines.  Some  micros,  like  the 
2901  Series,  can  be  used  in  2-  or  4-bit  slices.  It  is  not  necessarily  harder  to  troubleshoot 
them  than  other  digital  logic  devices. 

Embedded  computers  has  a  different  meaning  than  that  given  in  the  guide,  which 
seems  to  distinguish  between  embedded  computers  and  other  digital  devices.  A  better 
distinction  would  be  between  analog  and  digital  devices.  The  guide  distinction  between 
microprocessors  and  nonmicroprocessors  is  an  illogical  way  of  partitioning  the  world  of 
digital  devices. 

Embedded  computers  are  beneficial  or  adverse,  depending  on  the  nature  of  the 
system  and  the  implementation.  A  preferred  implementation  is  "modularity-isolated,  off¬ 
line,  used  for  automatic  performance  monitoring  or  built-in  test  and  automatic  fault 
localization,  and  fail-safe." 

5.  Design  Concepts  12  through  14  have  missed  the  point.  Standardization  per  se  has 
no  impact  on  manpower.  This  set  of  design  concepts  can  be  replaced  by  consideration  of 
spares  availability. 

6.  Design  Concept  15,  Standard  Hardware-Subsystems,  is  the  only  standardization 
decision  that  has  effects  on  human  resources  because  it  reduces  the  need  for  training. 
But  this  is  a  Navy  decision,  not  a  contractor  decision.  In  this  case,  the  acquisition 
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manager  needs  to  do  more  work  up  front  to  make  these  decisions  properly  before  he  lets 
out  the  specification.  In  order  to  do  this,  the  acquisition  manager  needs  technical 
assistance  which  he  doesn't  usually  have. 

7.  Design  Concept  17,  Built-in  Operator  Performance  Aids,  may  give  you  a 
marginal  gain  in  operator  performance  but  it  is  bought  at  the  expense  of  needing  a  high- 
level  maintainer  skill.  Thus,  the  overall  design  concept  is  not  all  that  desirable  unless  it  is 
required  for  safety. 

8.  Historically,  Design  Concept  18,  Automated  Decision-making,  is  only  imple¬ 
mented  in  a  halfhearted  way.  Unless  fully  automated  (eliminating  the  need  for  operators), 
this  concept  should  be  implemented  only  to  enhance  operator  display  information,  not 
make  decisions  for  him. 

9.  There  should  be  a  constraint  on  Design  Concept  20,  Built-in  Training  Capability, 
to  use  it  only  when  no  additional  hardware  is  required. 

10.  Design  Concept  21,  Combined  Operator/Maintainer  Functions,  is  okay  under 
several  circumstances  where  either  the  operator  or  the  maintainer  functions  are  so 
simplified  that  they  can  be  assumed  by  the  other. 

Additional  Design  Concepts.  An  additional  design  concept  is  the  tradeoff  between 
digital  and  analog  implementation  of  a  function.  Digital  functions  usually  require  fewer, 
often  no  adjustments;  digital  functions  are  easier  to  BITE  and  performance-monitor;  and 
digital  hardware  is  easier  to  standardize.  On  the  other  hand,  analog  functions  are 
traditionally  less  complex  in  parts  count  and  are  simpler  to  teach  and  understand; 
however,  these  advantages  are  rapidly  disappearing.  The  primary  advantage  of  analog  is 
the  ease  of  interfacing  to  the  external  world,  which  is  normally  also  analog.  Because 
digital  functions  are  complex  to  properly  implement  in  hardware,  software,  and  firmware, 
they  suffer  certain  disadvantages. 

An  additional  design  concept  would  be  to  consider  the  tradeoff  between  number  of 
cards  and  costs.  This  probably  has  a  manpower  implication.  This  is  one  of  the  practical 
problems  that  faces  the  engineer— how  big  should  the  cards  be  and  how  many  of  them 
should  there  be? 

General  Comments.  The  guide  could  help  us  by  telling  us  what  makes  a  system 
complex  to  an  operator  or  a  maintainer.  As  engineers,  we  act  on  intuition  and  common 
sense.  The  guide  could  help  us. 

Section  1  is  particularly  well  done  and  understandable.  However,  I  strongly  suggest 
that  you  add  at  least  one  example  under  each  definition. 

The  judges'  group  should  have  a  broader  experience  base  and  not  use  their  unsub¬ 
stantiated  feelings  as  a  basis  of  judgment.  The  group  might  be  expanded  prior  to 
publication. 

I  would  like  to  see  the  judges'  credentials,  perhaps  in  an  appendix. 

Because  the  evaluative  profiles  in  this  section  could  have  substantial  impact  on  the 
design  of  new  systems,  it  is  imperative  that  the  projected  impacts  be  quantitatively 
verified. 

The  treatment  of  the  design  concepts  as  all-or-nothing  or  go/no-go  factors  is 
unrealistic.  For  example,  there  are  degrees  of  "operational  simplicity." 
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The  design  concepts  presented  in  the  guide  are  extremely  heterogeneous  in  com¬ 
plexity.  It  would  be  helpful  to  break  the  design  concepts  down  into  logical  units  that  are 
more  equivalent  in  their  scope  and  complexity. 

You  must  state  the  constraints  for  all  the  design  concepts.  Tell  under  which 
circumstances  a  design  concept  is  beneficial;  tell  under  which  circumstances  it  should  not 
be  used.  Without  such  constraints,  somebody  who  is  writing  the  specifications  could  call 
for  the  design  concepts  in  the  guide  willy/nilly  and  the  contractor  would  be  stuck  with 
complying  with  something  that  could  be  inappropriate  and  even  silly. 

Section  2:  Interaction  of  Design  Concept  Impacts  on  Different  System  Design  Criteria 

The  content  of  Section  2  was  understandable  and  relatively  useful  to  most 
respondents.  No  consistent  themes  were  revealed  by  supporting  comments. 

Accuracy  and  completeness  of  this  section  was  considered  to  be  adequate,  but  was 
not  highly  rated.  Typically,  evaluation  of  accuracy  and  completeness  was  rated  as  2  or  3 
on  the  5-category  scales,  with  very  few  1  or  5  ratings.  Of  the  few  comments  made,  most 
questioned  the  qualifications  and  representativeness  of  the  sample  of  judges,  or  disagreed 
with  the  judges  on  specific  issues. 

Although  the  distribution  of  ratings  indicated  that  many  respondents  would  like  to  see 
changes  made  in  this  section,  there  was  no  consensus  regarding  these  changes.  A  wide 
variety  of  specific  problems  were  identified  and  suggestions  presented. 

Data  from  Six  Questions  in  Evaluation  Packages 

Figure  4  shows  how  respondents  answered  questions  regarding  the  information  in 
Section  2.  Comments  relative  to  these  questions  are  provided  below. 

1,  Useful  for  Present  Job?  (7) 

A  positive  comment  stated:  The  information  provided  a  good  summary  of  the 
problems  that  have  to  be  considered  and  covered  in  the  design. 

Three  negative  statements  included:  Section  2  is  a  repeat  of  Section  1  and  as 
such  is  of  little  value.  Would  not  know  how  to  use.  There  is  not  an  obvious  way  to  relate 
this  information  to  other  design  criteria. 

Two  commentators  suggested:  Needs  to  be  valid.  Another  input  should  be  made 
to  quantify  the  variables;  I  don't  feel  the  spread  is  adequate  and  the  answers  would  be 
different  for  different  types  of  equipment. 

One  comment  stated:  I  take  exception  to  the  adverse  effects  of  microcomputers 
on  new  systems.  If  1  followed  the  information  presented  here,  1  would  have  to  design 
around  it. 

2.  Useful  for  Serious  Manpower  Constraints?  (11) 

Many  of  the  comments  echoed  those  made  on  the  previous  question. 

Comments  included:  Useful  as  information  only.  More  positive,  explicit  data 
would  be  more  convincing.  More  of  a  hindrance  than  a  help;  it  would  be  the  basis  of  many 
meaningless  meetings.  Tradeoffs  should  be  made  by  the  offerer  relative  to  DTUPC  goals. 
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1.  USEFUL  FOR 
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Figure  4.  Ratings  of  information  in  Section  2:  Interaction  of  Design 
Concepts  on  Different  System  Design  Criteria. 
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The  data  provided  are  difficult  for  the  designer  to  quantify;  there  is  a  good  chance  the 
designer  will  be  confused  as  to  which  items  have  priority.  Perhaps  it  should  be  arranged 
in  order  of  Navy  equipment  operators,  then  ranked  in  order  of  Navy  equipment 
maintainers. 

One  respondent  made  the  following  tongue-in-cheek  comment:  There  are  no 
tradeoffs.  Just  follow  the  yellow  brick  road  as  dictated  by  manpower  constraints.  As  an 
example,  a  shipboard  system  can  be  designed  that  requires  "0"  shipboard  maintainers,  but 
the  costs,  space,  and  weight  may  well  be  impractical. 

3.  Accurate?  (12) 

Many  of  the  comments  reflected  those  that  were  made  under  the  question  of 
accuracy  in  Section  1.  These  comments  focused  on  the  qualifications  of  the  judges,  their 
number,  and  a  desire  to  see  the  data  base  validated.  Some  of  the  comments  simply 
expressed  disagreement  with  the  judges  on  several  issues. 

One  respondent  found  the  information  to  be  accurate,  with  the  exception  of 
"embedded  computers."  He  suggested  that  new  data  be  acquired  from  operators  and 
maintainers  that  have  systems  that  contain  embedded  computers. 

One  respondent  stated:  The  profile  for  Criterion  M  warps  the  emphasis  on 
"initial"  as  opposed  to  life  cycle  costs,  which  is  the  purpose  of  the  manual.  You  should 
interchange  Profiles  M  and  N. 

4.  Complete?  (4) 

Comments  included:  Criteria  should  be  more  definitive.  Information  is  very 
generalized;  possibly  too  complex  for  small  systems.  Is  it,  in  fact,  personal  observation? 
Design  Concepts  8,  9,  10,  and  1 1  presuppose  that  embedded  computers  will  not  be  used  for 
these  concepts,  or  else  there  is  a  built-in  conflict. 

5.  Understandable?  (8) 

Three  comments  concerned  the  relationship  between  Sections  1  and  2.  A  bit 
misleading  at  first;  this  chapter  should  be  described  as  a  simple  cost-indexing  of  Section 
1,  not  as  "complementary"  to  Section  1,  implying  new  information.  Should  reference  back 
to  Figure  3;  a  heavier  black  outline  around  the  second  box  in  Figure  5  would  help; 
establish  a  clear  relationship  to  Section  1.  Your  technique  of  interchanging  the  ordinate 
and  abscissa  between  Sections  1  and  2  is  poor;  I  was  prepared  to  follow  the  structure  of 
Section  1  into  Section  2. 

Two  comments  concerned  definition  of  the  system  design  criteria:  I  believe  that 
the  explanations  of  the  evaluation  criteria  could  be  improved;  for  example,  I  don't 
understand  the  meaning  of  the  last  sentence  on  Page  2-4.  The  criteria  in  Section  2  were, 
in  general,  not  clearly  stated. 

Two  comments  indicated  insufficient  direction  on  how  to  use  the  information  in 
Section  2:  The  charts  are  subjective  and,  although  they  provide  interesting  data,  leave 
the  designer  with  questions  as  to  which  design  direction  should  be  followed.  There  is  not 
enough  instruction  given  to  indicate  how  to  use  the  information. 

One  commentator  thought  that  Section  2  was  "very  pictorial." 
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6. 


Should  be  Changed?  (14) 


Several  comments  restated  comments  made  earlier  and  will  not  be  repeated. 
Short  comments  included:  Should  be  quantified  and  backed  up.  Omit  criteria  not 
considered  human  resources  (e.g.,  costs,  MTBF).  Provide  a  single  table  showing  numerical 
values. 


Two  comments  focused  on  applying  the  information:  There  need  to  be  examples 
of  how  to  use  the  information;  need  a  roadmap  through  the  material.  Clarify  definitions 
of  the  criteria  and  give  one  or  more  examples  of  the  application  of  each  criterion. 

Two  comments  focused  on  the  interaction  among  the  criteria:  It  is  difficult  to 
balance  all  the  factors  involved  in  order  to  decide  which  set  of  criteria  should  be 
emphasized.  More  specifics  are  needed  on  the  interaction  of  the  criteria. 

Two  comments  focused  on  questions  of  specific  equipment:  It  would  appear  that 
the  section  would  have  to  be  tailored  to  meet  the  equipment  type  and  mission  as  the  Navy 
sees  it;  summarize  Section  2  with  a  designer’s  checklist.  Information  must  be  specifically 
related  to  system  size/complexity  and  unit  levels. 

One  commentator  opined:  It  seems  that  at  this  point  you  were  more  interested 
in  presenting  "variations  on  a  theme"  for  academic  interest  and  statistical  analysis  than 
you  were  in  developing  a  good  solid  rationale  which  can  build  a  basis  for  Sections  5  and  6. 

Data  on  Specific  System  Design  Criteria 

Table  2  shows  how  respondents  rated  the  understandability  and  utility  of  the  system 
design  criteria.  As  with  the  design  concepts,  the  percentages  are  based  upon  the  number 
who  responded  under  each  category.  Forty  to  43  respondents  rated  the  understandability 
of  the  design  criterion;  and  36  to  42,  their  utility.  As  reflected  in  the  table,  respondents 
had  no  problem  in  understanding  the  criteria,  and  few  proposed  dropping  any  of  them. 
Five  system  design  criteria — A,  B,  D,  G,  and  3- -were  rated  as  useful  but  in  need  of 
modification  by  25  percent  or  more  of  the  respondents. 

Comments  relative  to  the  specific  design  criteria  are  provided  in  the  following 
paragraphs. 

1.  A--Maintainer  Skill  and  Experience  Level  Required.  (12) 

Five  comments  concerned  the  definition  and  scope  of  the  criterion:  The  text 
assumes  that  every  designer  understands  the  skill  level  of  "first  enlistment"  and  "second 
enlistment"  personnel;  a  summary  of  skill  levels  could  be  added.  Don't  understand 
meaning  of  last  sentence  in  definition.  Guidance  is  needed  in  associating  maintainer's 
skill  with  rating/training,  e.g.,  what  are  the  percentages  of  faults  in  representative 
hardware  which  can  be  repaired  by  various  types  of  personnel;  can  the  proficiency  profiles 
from  Section  6  be  made  more  quantitative?  I  don't  think  the  skill  levels  should  be  limited 
to  just  "long  timers,"  but  should  also  indicate  new  enlistee  capabilities.  It  would  be  useful 
to  know  the  feelings  about  the  percentage  change  of  those  needed  beyond  the  first 
enlistment;  these  charts  only  consider  about  one-half  the  problem. 

Two  comments  focused  on  embedded  computers:  I  don't  agree  with  the  judges  on 
embedded  computers.  New  designs  will  definitely  contain  microprocessors;  therefore,  a 
maintainer  will  master  this  "new"  device  such  that  it  will  eventually  be  nothing  more  than 
a  "logic  board"  of  future  systems. 
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Table  2 


Reactions  of  Commentators  to  Specific  System  Design  Criteria 


Understandable  Utility 

(%  Responding)  (%  Responding) 


DESIGN  CONCEPT 

I  Under¬ 
stand  it 

I  Don't 
Under¬ 
stand  it 

USEFUL: 

Keep 

it 

USEFUL: 

Modify 

it 

USELESS: 

Drop 

it 

A.  Maintainer  skill  and  experience  level 
required 

98 

2 

69 

29 

2 

B.  System-specific  maintenance  training 
required 

98 

2 

70 

28 

2 

C.  Shipboard  maintenance  man-hours  required 

100 

0 

82 

15 

3 

D.  MTBF 

100 

0 

70 

25 

5 

E.  MTTR 

100 

0 

72 

23 

5 

F.  Tools,  test  equipment,  facilities  costs 

100 

0 

80 

15 

5 

G.  Supply  and  support  costs 

100 

0 

64 

31 

5 

H.  Operator  skill  and  experience  level  required 

100 

0 

82 

15 

3 

I.  System-specific  operator  training  required 

100 

0 

85 

12 

3 

3.  Total  number  of  operators  required 

100 

0 

72 

25 

3 

K.  System  operability 

98 

2 

78 

19 

3 

L.  Overall  operational  capability  and 
effectiveness 

95 

5 

68 

24 

8 

M.  Initial  system  acquisition  costs 

100 

0 

81 

14 

5 

N.  Operational  lifetime  costs 

100 

0 

72 

22 

6 

Note.  20  percent  of  the  respondents  indicated  that  additional  system  design  criteria  should  be  considered. 


A  variety  of  critical  comments  included:  Don't  agree  with  7,  18,  19,  and  20; 
these  are  beneficial,  not  adverse.  Design  Concepts  2  and  3  should  be  more  negative;  5 
should  be  more  positive;  7  should  be  much  more  positive.  Design  Criteria  A  and  B  should 
be  combined.  The  criteria  and  the  topic  need  to  be  reexamined. 

2.  B- -System-Specific  Maintenance  Training  Required.  (9) 

A  variety  of  short  comments  offered:  What  is  the  magnitude  of  system 
complexity  (e.g.,  hardware,  software,  firmware)?  If  you  are  using  current  NECs  to 
maintain  new  systems,  then  the  entire  reference  moves  toward  adverse.  The  designer 
probably  will  not  know  the  training  time  requirements.  Recognize  the  advent  of  LSI, 
VLSI,  and  distributed  computation  on  future  training  requirements.  Criteria  are  not 
clearly  stated.  Agree  on  embedded  computers;  these  should  be  as  easy  to  maintain  as  any 
other  logic  elements.  Combine  this  criterion  with  Criterion  A. 

Two  comments  focused  on  standard  hardware:  Standard  hardware  components 
would  have  a  positive  impact  on  system  specific  maintenance  training  only  if  onboard 
repair  is  required.  Do  you  consider  standard  building  hardware  (e.g.,  UYK-20  and  UYQ- 
21)  system  specific?  Generally,  there  does  not  seem  to  be  enough  emphasis  on  this  design 
feature. 


3.  C — Shipboard  Maintenance  Man-hours  Required.  (3) 

Two  comments  concerned  embedded  computers:  Disagree  on  embedded 

computers.  The  maintenance  of  embedded  computers  is  dependent  on  the  maintenance 
philosophy  employed  and  may  not  necessarily  be  as  negative  as  shown. 

One  commentator  suggested  that  the  criterion  be  split  into  preventive 
maintenance  and  corrective  maintenance. 

4.  D--MTBF.  (8) 

One  comment  indicated  that  the  presentation  of  data  confused  cause  and  effect: 
MTBF  is  a  result  of  the  design  concept  and  not  the  cause  of  any  adverse  or  beneficial 
effect. 


Two  comments  discussed  dropping  or  modifying  the  criterion:  Combine  MTBF 
and  MTTR  which  then  become  "corrective  maintenance";  also,  MTBF  is  too  pure  a  concept 
to  be  practically  applied  here  as  it  is.  Drop  MTBF  and  MTTR  because  they  are  adequately 
covered  already  by  reliability  and  maintainability  specifications  and  organizations. 

A  variety  of  disagreements  with  the  data  included:  The  profile  is  not  reasonable 
for  built-in  test  equipment,  which  should  be  desensitized  to  MTBF;  it  should  not  cause 
system  failure;  also,  need  to  define  MTBF  more  specifically.  I  question  the  results  for 
Design  Concept  6.  I  do  not  believe  that  performance  monitoring,  fault  detection,  and 
fault  localization  equipment  should  degrade  system  performance  (MTBF)  as  indicated; 
better  approaches  are  available.  As  an  LSI  circuit  itself,  embedded  computers  have  a 
very  high  MTBF  and  will  not  degrade  a  system  (as  suggested  by  the  guide):  MTBF  may 
actually  improve  when  you  consider  the  amount  of  hardware  an  embedded  computer 
replaces! 


One  commentator  offered  the  following  analysis:  I  disagree  with  the  relationship 
between  MTBF  and  Design  Concept  7,  8,  11,  and  19.  Do  you  distinguish  between  MTBF 
and  MTBE?  With  good  design,  a  test  system  will  not  impact  MTBF,  but  may  contribute  to 
MTBE;  E  stands  for  "events,"  which  are  not  impacts  on  performance. 
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5.  E--MTTR.  (6) 


One  comment  indicated  confusion  of  cause  and  effect  in  System  Design  Criterion 
D:  MTBF  is  a  result  of  the  design  concept  and  not  the  cause  of  any  adverse  or  beneficial 
effect. 


Two  commentators  disagreed  on  the  results  for  Design  Concept  6:  Overdesign 
will  result  in  potentially  fewer  provisioned  spares;  in  the  event  of  failure,  MTTR  may 
actually  be  adverse  rather  than  "0."  Disagree  on  Item  6;  it  takes  longer  to  repair  items 
that  you  are  not  familiar  with  (have  had  no  practice  on). 

A  variety  of  other  disagreements  with  the  data  included:  embedded  computers 
reduce  MTTR.  LRUs— No  Spares  must  have  an  adverse  effect!  Since  the  concept  of  using 
standard  hardware  is  that  circuit  cards  utilize  throwaway  maintenance,  one  might  expect 
that  Concept  13  on  the  graph  would  be  nearly  as  positive  as  Concept  5. 

6.  F — Tools,  Test  Equipment,  Facilities  Cost.  (5) 

One  comment  indicated  confusion  with  respect  to  the  presentation  of  informa¬ 
tion:  Built-in  Test  Equipment  was  not  to  be  included,  according  to  the  definition,  but  it 
was  estimated  to  have  a  15  percent  beneficial  effect. 

One  commentator  suggested  that  this  design  criterion,  along  with  G,  M,  and  N, 
be  dropped  because  they  are  "adequately  covered  by  cost  specifications  and  organiza¬ 
tions."  Another  commentator  believed  that  System  Design  Criterion  F  should  be 
combined  under  a  single  heading  concerned  with  nonrecurring  costs. 

One  commentator  disagreed  with  the  data  on  Design  Concept  6:  With  over- 
design,  test  equipment  may  not  be  available.  This  would  adversely  affect  MTTR. 

One  comment  suggested  a  need  for  additional  data:  As  a  designer,  I  would  have 
a  difficult  job  in  making  an  estimate  for  the  tender  and  depot  costs  for  test  equipment 
and  facilities.  Either  some  additional  data  should  be  provided,  or  the  tender  and  depot 
aspects  of  this  criterion  should  be  dropped. 

7.  G — Supply  and  Support  Costs.  (10) 

One  of  the  comments  could  not  be  interpreted.  Suggestions  included:  This  is 
very  difficult  to  quantify  but  it  should  be  pursued;  these  secondary  effects  are  of  a  most 
serious  magnitude.  You  should  split  depot,  tender,  and  shipboard  costs  into  separate 
items.  As  a  designer,  1  would  have  a  difficult  job  in  estimating  costs  for  cataloging, 
receipt,  storage,  transfer,  issue,  etc.;  some  additional  guides  have  to  be  provided.  If  life 
cycle  cost  studies  were  a  condition  of  the  contract,  the  Navy  should  supply  a  set  of 
standard  values  to  establish  a  common  baseline. 

Disagreements  with  the  data  included:  Design  Concept  7  should  be  more 
positive;  Design  Concept  20  should  be  more  negative.  This  curve  is  misleading;  benefits 
are  indicated  from  standard  hardware,  but  that  is  an  oversimplification  when  considering 
overall  costs. 

Three  disagreements  focused  on  repair:  1  believe  contributors  to  the  study  were 
truly  off  base  on  Design  Concept  5;  were  they  knowledgeable  in  costs  of  repair  versus 
throwaway?  It  is  my  understanding  that  the  decision  to  use  throwaway  maintenance  is 
based  on  a  unit  price  cost  where  it  is  cheaper  to  buy  a  new  unit  than  to  repair  it; 
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therefore,  I  do  not  understand  why  throwaway  maintenance  had  such  a  negative  impact  on 
supply  and  support  costs.  1  believe  the  judges  did  not  account  for  the  inventory  and 
distribution  costs  for  onboard  repair. 

8.  H — Operator  Skill  and  Experience  Level  Required.  (4) 

Three  of  the  comments  suggested  that  there  was  a  problem  in  understanding  this 
design  criterion.  Part  of  the  confusion  focused  on  the  meaning  of  the  term  "number  of 
people  beyond  their  first  enlistment  required  to  ensure  a  high  degree  of  system 
operational  effectiveness."  One  comment  suggested  that  this  criterion  be  combined  with 
Criteria  I  and  K  and  presented  on  one  graph. 

9.  I — System-specific  Operator  Training  Required.  (2) 

One  of  the  comments  suggested  difficulty  in  understanding  the  criterion.  The 
other  comment  suggested  that  it  be  combined  with  Criterion  H,  because  the  judges’  data 
showed  that  there  was  little  difference  in  the  judges'  perception  of  the  two  criteria. 

10.  3 — Total  Number  of  Operators  Required.  (8) 

Several  of  the  comments  indicated  that  there  was  confusion  about  the  definition 
and  scope  of  Design  Criterion  J.  Some  of  these  comments  indicated  that  the  criterion 
should  be  modified  to  be  more  clear. 

One  respondent  stated  that  embedded  computers  and  automatic  performance 
monitoring  should  have  a  beneficial  impact  on  the  number  of  operators. 

One  commentator  stated:  This  analysis  doesn't  address  the  changing  demands  on 
Operational  Readiness  Operators  (e.g.,  the  assessment  of  workload). 

Two  comments  suggested  additional  design  criteria,  which  will  be  treated  in  a 
later  section. 

11.  K --System  Operability.  (4) 

Two  of  the  comments  addressed  quantification  problems:  Operator  error  rates 
will  depend  on  each  individual  operator  and  thus  are  very  difficult  to  estimate.  It's  very 
difficult  to  quantify  this  criterion  but  it  should  be  pursued;  these  secondary  effects  of 
system  design  are  of  most  serious  magnitude. 

Two  of  the  comments  focused  on  the  definition  of  the  criterion:  1  think  ease  of 
operation,  error  rate,  and  reaction  time  should  be  independent  criteria.  Consider  adding 
system  flexibility  and  degraded  mode  operation  to  the  definition  of  system  operability. 

12.  L— Overall  Operational  Capability  and  Effectiveness.  (8) 

Four  comments  focused  on  problems  with  the  definition  and  scope  of  the 
criterion:  I  think  more  specificity  is  required;  several  broad  operational  requirements 
(e.g.,  system  reaction  time)  should  be  cited  and  rated  separately.  The  definition  is  too 
all-encompassing;  there  is  nothing  specific  to  grasp  as  a  reference  for  what  you  mean.  It's 
too  judgmental;  you  need  more  specific  criteria  to  be  useful.  A  measure  of  equipment 
effectiveness  would  have  to  be  tailored  to  the  specified  mission  requirements;  it  is  not 
clear  what  parameters  were  used  for  this  measurement.  One  commentator  said:  the 
concept  of  technical  performance  should  be  tied  to  operational  performance  in  some 
manner;  there  are  tradeoffs  between  these  two  which  could  be  addressed. 
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One  commentator  repeated  his  earlier  comment:  It  would  be  very  difficult  to 
quantify  but  it  should  be  pursued;  these  secondary  effects  of  system  design  are  most 
serious  in  magnitude. 

One  commentator  said:  This  is  confusing;  I  do  not  understand  why  all  are  judged 
beneficial. 

One  commentator  suggested  the  criterion  be  dropped:  This  is  probably 
adequately  covered  by  system  performance  specifications  and  acceptance  testing. 

13.  M--lnitial  System  Acquisition  Costs.  (3) 

One  commentator  stated:  As  a  designer,  I  would  have  trouble  determining  the 
increase  or  decrease  of  these  costs.  I  would  need  some  backup  data. 

Another  stated:  The  perceptions  of  your  sample  group  are  biased  due  to  their 
backgrounds.  From  the  designer's  standpoint,  several  items  do  not  increase  initial  costs  if 
they  are  incorporated  early  and  if  the  designer  is  accustomed  to  the  concept  (e.g.,  1,  2,  9, 
and  10).  Other  concepts  will  increase  costs  due  to  the  rapid  movement  of  technology 
(e.g.,  12,  13,  14,  and  15). 

One  commentator  suggested  that  the  overall  presentation  would  be  improved  if 
Profiles  M  and  N  were  exchanged  in  order  of  presentation.  He  also  stated:  The  profile 
for  Criterion  M  is  misleading-benefits  are  indicated  from  standard  hardware,  but  this  is 
an  oversimplification  when  considering  overall  costs. 

14.  N — Operational  Lifetime  Costs.  (5) 

Three  of  the  comments  were  reiterations  of  remarks  concerning  other  criteria. 
They  were:  Very  difficult  to  quantify  but  should  be  pursued;  the  secondary  effects  are  of 
a  most  serious  magnitude.  As  a  designer,  I  would  have  trouble  determining  the  increase 
or  decrease  of  these  costs;  I  would  need  some  backup  data.  The  perceptions  of  your 
sample  group  are  biased  due  to  their  backgrounds;  from  the  designer's  viewpoint,  several 
items  do  not  increase  initial  costs  when  incorporated  early  and  when  the  designer  is 
accustomed  to  the  concept  (e.g.,  Concepts  I,  7,  9,  and  10),  other  concepts  will  increase 
costs  due  to  the  rapid  movement  of  technology  (e.g.,  12,  13,  14,  and  15). 

New  thoughts  include:  Would  prefer  some  relationships  to  provide  hard  tradeoff 
decisions  between  life  cycle  costs  and  acquisition  costs  to  make  an  intelligent  and  proper 
decision.  I  do  not  agree  with  judges  on  the  adverse  impact  of  Design  Concepts  4  and  5 
versus  the  "0"  impact  of  Concepts  2  and  3. 

Suggested  Additional  System  Design  Criteria 

Fourteen  additional  system  design  criteria  were  suggested  by  the  commentators.  One 
was  uninterpretable;  one  was  actually  an  additional  design  concept,  which  was  treated 
earlier. 

One  commentator  suggested  that  Design  Criterion  3,  total  number  of  operators 
required,  actually  depended  on  the  system  mission  requirements.  He  suggested  that  a 
better  criterion  would  be:  Total  number  of  operator  interfaces,  which  would  relate  to 
operator  station  complexity. 

Another  commentator  suggested  that  the  minimum  number  of  required  operators 
would  be  a  good  criterion  in  that  it  would  be  independent  of  any  baseline. 
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One  commentator  said  that  watch  station  requirements  under  Conditions  1  and  3 
would  be  a  good  design  criterion. 

One  respondent  wrote  that  Mean  Time  Between  Removals  (MTBR)  would  be  an 
important  criterion  for  maintainability,  particularly  where  fault  isolation  or  BITE  is 
incorporated.  Another  commentator  believed  that  Mean  Logistics  Down  Time  (MLDT)  is 
the  criterion  that  really  drives  system  availability. 

One  commentator  wanted  to  lump  several  classic  criteria  under  one  heading:  The 
topic  of  Operational  Availability  or  Dependability  might  provide  a  more  effective 
measurement  of  the  interface  between  MTBF,  MTTR,  Spares  Availability,  Spares  Repair 
Time,  etc. 

One  commentator  suggested  that  the  ability  to  modify  and  upgrade  a  system  is  one 
important  design  criterion. 

One  commentator  believed  that  the  requirements  for  documentation  (e.g.,  training 
manuals,  drawings,  etc.)  is  an  important  design  criterion. 

One  commentator  appeared  to  want  to  refine  some  aspect  of  System  Design  Criterion 
N,  Operational  Lifetime  Costs:  How  about  life  cycle  costs?  Logistic  elements  seem  too 
vaguely  employed. 

Two  commentators  focused  on  support  maintenance.  One  stated:  Total  maintenance 
impact  involves  shore  maintenance;  I  believe  some  reference  should  be  made  to  IMA 
effort;  shipboard  maintenance  goes  to  "0"  if  you  weld  the  thing  closed.  The  other  stated: 
How  much  non-Navy  support  is  required,  e.g.,  how  much  support  from  NAVSEACENLANT, 
NAVSEACENPAC,  Contractor,  and  ISEA? 

General  Comments  on  System  Design  Criteria 

The  following  paragraphs  contain  comments  without  introductory  preamble. 

1  understand  what  you  are  driving  at  but  I  don't  know  how  to  achieve  a  proper  balance 
in  my  own  mind.  Different  customers  react  in  different  fashions  and  many  people  tend  to 
emphasize  their  own  pet  prejudices.  As  an  example,  operational  lifetime  costs  would  be 
important  over  a  long  haul,  but  the  initial  buyer  is  interested  in  the  initial  system 
acquisition  costs.  I  don't  think  your  guide,  or  anybody  else's,  can  resolve  the  dilemma. 

Given  the  definition  of  embedded  computers,  I  do  not  agree  with  the  judges' 
evaluation.  Their  collective  judgments  are  political  and  lacking  in  understanding  of  what 
can  be  done. 

This  is  merely  another  permutation  of  the  bad  data  contained  in  Section  1  and  the 
quantification  is  therefore  questionable.  The  categories  are  valid,  but  the  use  of  them  is 
not.  For  instance,  opinion  cannot  tell  if  use  of  embedded  processors  will  drastically 
affect  lifetime  cost;  it  will  vary  between  systems. 

Basic  areas  and  concepts  are  valid,  but  they  are  not  applied  correctly.  Most  of  these 
areas  require  a  good  deal  of  1LS  analysis  arid  should  be  addressed  that  way  for  varying 
implementations. 

This  section  presents  interesting  information,  but  how  does  a  design  engineer  use  it? 


Ir 
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In  general,  I  feel  that  all  of  the  descriptions  could  be  beefed  up  and  expanded. 

Comments  from  Other  Sources 

All  of  the  comments  from  other  sources  reiterated,  in  one  way  or  another,  comments 
from  the  Evaluation  Packages  C  and  G.  They  will  not  be  repeated  here. 

Section  3:  Types  of  Technicians  Assigned  to  Surface  Ship  Electronic  Systems 

Although  respondents  had  little  trouble  in  understanding  the  information  in  Section  3, 
many  of  them  questioned  its  accuracy  and  completeness.  As  a  consequence,  respondents 
varied  greatly  in  how  useful  they  considered  Section  3  to  be  for  their  present  job  or  for 
the  imposition  of  design  constraints  on  manpower  considerations.  Nearly  all  the 
suggestions  for  change  were  to  expand  the  information  contained  in  Section  3.  The 
following  specific  additions  were  recommended:  more  complete  profile  of  operators  and 
maintainers,  and  procedures  for  better  relating  technician  information  to  future  trends  in 
equipment  design. 

Data  from  Six  Questions  in  Evaluation  Packages 

Figure  5  shows  how  respondents  answered  questions  regarding  the  information  in 
Section  3.  Comments  relative  to  these  questions  are  provided  below. 

1.  Useful  for  Present  Job?  (13) 

Most  comments  focused  on  the  inadequate  amount  of  material  presented  on  this 
topic:  Just  doesn't  tell  me  much.  Information  is  very  general  and  could  be  expanded 

upon  to  be  more  useful.  Not  enough  information  to  be  very  useful;  the  material  is  too 
blah. 

There  were  a  number  of  suggestions  for  improving  this  section:  A  better 
description  of  skills  and  skill  levels  is  appropriate.  1  could  use  a  list  of  courses  and 
materials  in  the  courses  as  well  as  the  general  education  and  skill  levels  of  the  various 
ratings.  Would  be  useful  to  include  a  profile  of  the  source  of  training  taken  by  each  of  the 
ratings. 


One  respondent  said:  This  section  deals  with  a  critical  point  with  which 
designers  are  perhaps  least  familiar— the  job  description  approach  seems  totally  in¬ 
adequate--!  feel  this  aspect  of  the  handbook  should  be  expanded  significantly,  identifying 
the  training  requirements,  typical  time  on  job,  I.Q.  spread,  etc.;  lay  it  on  the  line  as  to 
what  the  problem  really  is. 

One  commentator,  a  member  of  the  acquisition  community,  said:  The 

engineer/designer  doesn't  give  a  jolly  damn  about  the  job  title  of  the  man  who  is  supposed 
to  fix  the  gear.  The  personnel  specialist  on  the  project  knows  more  about  Rating 
Descriptions  than  is  offered  here. 

An  engineer  said:  I  assume  that  the  lowest  possible  level  of  technician  will  be 
available  and  assigned  for  equipment  service/operation. 

One  commentator  described  a  future  problem:  Integrated  and  all-digital  systems 
will  not  permit  distinctions  between  certain  ratings  that  have  historically  assumed 
different  portions  of  system  maintenance.  They  must  now  be  lumped  together. 
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Figure  5.  Ratings  of  information  in  Section  3:  Types  of  Technicians 
Assigned  to  Surface  Ship  Electronic  Systems. 
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Useful  for  Serious  Manpower  Constraints?  (5) 


Many  of  the  comments  on  this  question  referred  back  to  those  on  the  previous 
question.  Respondents  said  that  the  same  drawbacks  of  the  material  with  regard  to 
usefulness  in  their  present  jobs  would  obtain  in  future  attempts  to  impose  serious 
manpower  constraints.  Typical  comments  were:  The  information  is  too  superficial  to  be 
of  any  but  token  value.  Not  enough  information  to  be  very  useful. 

Echoing  an  earlier  concern  regarding  the  future  of  the  Navy  rating  structure,  one 
commentator  said:  The  information  is  useless— the  Navy  structure  is  too  traditional  and 
directed  toward  outdated  equipment  and  not  the  maintenance  of  modern  integrated 
systems. 


3.  Accurate?  ( 1 ) 

The  comment  was:  This  information  may  not  be  accurate  for  the  rating 
structure  planned  for  ships  that  are  currently  under  development.  The  information  in  this 
section  may  be  obsolete  before  the  report  is  even  published. 

4.  Completeness?  (14) 

Comments  on  completeness  echoed  those  on  usefulness.  One  commentator 
summed  it  up  succinctly  by  saying:  A  chart  that  says  sonar  technicians  repair  and  operate 
sonars  is  a  bit  much— surely  a  more  detailed  description  of  skill  levels  is  available. 

Other  comments  included:  Quantities  could  be  presented.  Information  on 
training  curriculums  could  be  presented.  There  is  not  enough  meat.  Skill  levels  within  the 
types  of  skills  were  glossed  over;  the  levels  of  skills  should  be  mentioned  so  that  this 
information  will  be  in  perspective  when  it  appears  in  later  sections  of  the  guide. 

Two  respondents  reiterated  the  concern  that  the  information  may  be  out  of  date. 
One  said:  The  information  is  20  years  out  of  date.  The  other  added:  For  those  types  of 
systems  in  which  standard  displays  are  shared,  a  maintenance  specialist  should  be  created 
to  take  care  of  the  standard  hardware,  regardless  of  the  traditional  rating  in  which  the 
hardware  may  fall. 

Two  commentators  noted  that  the  section  does  not  tell  how  to  determine  the 
grade  of  personnel  or  the  number  of  personnel  required  for  a  system  under  design  nor  does 
it  provide  a  detailed  procedure  to  compare  the  complexity  of  a  present  system  with  the 
complexity  of  a  past  system  so  that  one  could  judge  the  manpower  skill  required  for  the 
present  system. 

Three  commentators  suggested  that:  The  guide  should  have  a  broader  scope,  it 
should  cover  other  types  of  electronic  systems  and  other  types  of  platforms  to  be  more 
complete.  Elsewise,  the  title  of  the  guide  should  be  changed  to  specify  the  limitations  of 
its  applicability. 

5.  Understandable?  (4) 

There  were  no  comments  on  precisely  how  understandable  Section  3  was  to 
readers.  One  respondent  noted  that  designers  can  identify  required  quantities  of 
personnel  and  tasks  that  need  to  be  performed  and  that  they  probably  would  not  care  that 
the  personnel  titles  might  be  ET,  FT,  etc.  An  opposite  view  suggested  that  more  rather 
than  less  information  should  be  provided;  this  commentator  suggested  that  rating  profiles 
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for  each  technical  rating  be  provided  along  with  a  sample  shipboard  organization  chart 
showing  rating  structure. 

6.  Should  be  Changed?  (19) 

One  commentator  stated  that  the  coverage  of  personnel  types  as  presented  was 
good.  The  remaining  comments  on  this  question  reflected  a  need  for  more  detail:  Two 
pages  can't  answer  Question  3  as  the  guide  purports.  Either  drop  the  section  or  expand  it 
enormously.  Tell  me  something  useful  or  forget  it.  Expand  it.  The  information  should  be 
expanded  or  deleted.  You  don’t  expect  companies  with  no  background  to  participate  in 
this  business;  the  data  supplied  imparts  no  information.  It  doesn't  tell  much  about  skill 
levels;  it  is  not  possible  to  quantify  them,  given  the  individual  skill  variances. 

Suggestions  for  improvements  and  changes  in  this  section  include:  Could  use 
elaboration  of  equivalent  educational  levels  or  extent  of  training  and  average  years  in 
service.  Provide  a  better  profile  of  the  personnel  who  will  actually  be  performing  the 
work.  Need  more  details  to  understand  the  amount  of  training  and  requirements  for 
personnel  at  the  various  levels.  Should  add  educational  level  and  job  descriptions. 

Some  commentators  again  noted  the  problem  of  relating  information  about 
technicians  to  current  design  trends.  Their  comments  include:  In  current  system  design, 
the  roles  of  DS  and  FTM  have  tended  to  merge.  Multifunction  1980  radar  systems,  which 
combine  detection,  tracking,  evaluation,  weapon  assignment,  fire  control  solution,  missile 
launch,  midcourse  guidance,  and  kill  evaluation  on  multiple  targets,  have  equipment  and 
computer  configurations  which  require  a  "new  breed"  of  technicians.  Combat  control 
systems,  integrated  and  all-digital,  will  not  permit  distinctions  between  DS  and  FTM 
ratings  for  maintenance;  designers  cannot  conceptualize  certain  parts  of  the  design  as  for 
DS  and  other  parts  FTM. 

Other  respondents  suggested  relating  information  about  technicians  to  an 
analysis  of  the  equipment  complexity:  Need  a  more  analytical  approach  to  skill  levels  by 
either  complexity  factor,  number  of  components,  or  type  of  components,  etc.  Relate 
system  complexity  to  quantity  of  operators  and  maintainers. 

One  commentator  suggested  combining  this  section  with  Section  8,  which  covers 
training  requirements  and  Navy  Enlisted  Classifications  (NECs). 

Comments  from  Other  Sources 

These  comments  were,  to  a  certain  degree,  similar  to  information  from 
Evaluation  Packages  C  and  G. 

Three  commentators  from  the  acquisition  community  indicated  that  Section  3 
contains  information  that  is  useful  to  designers.  The  other  comments  were  to  the  effect 
that  the  information  was  interesting,  but  not  presently  useful  to  designers.  One 
commentator  stated:  The  information  is  superficial  and  should  have  been  integrated  into 
Section  4  as  part  of  the  definitions  of  the  types  of  personnel. 

Another  evaluator,  agreeing  with  sentiments  expressed  in  the  Evaluation  C  and  G 
data,  suggested  that  Section  3  be  expanded  to  include  rating  profiles,  departments  of 
shipboard  organization,  watch  assignments,  maintenance  tasks,  etc.  This  commentator 
also  suggested  that  a  brief  statement  of  personnel  utilization  during  Condition  1  and 
Condition  3  would  be  helpful. 


45 


Section  4:  Projected  Supply  of  Technical  Ratines  at  Different  Experience  Levels 


There  was  substantial  disagreement  among  respondents  as  to  how  useful  Section  4 
would  be  to  system  designers.  Ratings  on  usefulness  tended  to  be  bimodal.  Comments 
made  indicated  that,  on  the  negative  side,  the  information  was  not  relevant  to  system 
designers  and  their  design  task.  Some  went  further  to  state  that  the  supply-of-technicians 
problem  was  the  Navy's  to  solve,  not  the  designer's. 

Although  the  accuracy  and  completeness  of  the  information  in  Section  4  was  rated 
relatively  high  by  those  who  completed  ratings,  relatively  large  percentages  indicated 
they  did  not  know  enough  to  complete  a  rating. 

The  main  changes  recommended  were:  drop  the  information  because  it  is  not 
relevant,  and  expand  the  projections  to  be  consistent  with  20-year  system  life  cycles. 

Data  from  Six  Questions  in  Evaluation  Packages 

Figure  6  shows  how  respondents  answered  questions  regarding  the  information  in 
Section  4.  Comments  relative  to  these  questions  are  provided  below. 

1.  Useful  for  Present  Job?  (16) 

Positive  comments  included:  Liked  the  information  in  the  presentation.  Useful 
for  trade  studies  and  system  design  goals. 

Another  group  of  comments  suggested  that  the  information  was  not  relevant  to 
designers:  The  information  would  be  of  general  interest  only.  Information  is  nice  but  only 
serves  to  reinforce  problem  of  manning,  not  help  system  designer.  Personnel  availability 
is  of  interest  to  the  designer  only  as  a  point  of  emphasis  as  to  why  equipment  should  be 
designed  with  the  level  of  skill  and  manpower  established;  1  don't  feel  the  availability  of 
personnel  statistics  is  important.  None  of  this  section  tells  the  designer  much.  Designer 
does  as  good  a  job  as  he  can  within  the  constraints  given  him;  don't  see  how  this 
information  is  going  to  help  him.  All  systems  are  already  required  to  be  designed  for 
operation  and  maintenance  using  personnel  of  the  lowest  skill  level  practicable. 

Another  group  of  commentators  agreed  that  the  information  was  irrelevant  to 
designers,  but  added  the  opinion  that  responsibility  for  addressing  the  manpower  shortage 
problem  lay  elsewhere:  ,A II  this  says  is  that  there  will  be  certain  shortages;  maybe  it 
would  be  better,  cheaper,  to  generate  incentives  (within  the  Navy)  to  reduce  the  shortages 
than  to  add  cost  and  complexity  to  the  equipment.  This  is  not  the  designer's  problem  but 
the  Navy's;  why  do  only  one  in  five  sign  up  for  a  second  hitch  (motivation?);  why  are 
senior  rates  in  short  supply  (career-path  development  is  so  poor?).  The  problem  here  is 
obvious,  but  it  is  with  the  Navy  and  not  the  information;  the  Navy  must  broaden 
electronic  technician  training. 

One  commentator  echoed  the  notion  that  the  blurring  of  distinctions  between  DS 
and  FTM  ratings,  as  well  as  others,  makes  projected  supply  statistics  that  are  broken  down 
in  the  traditional  fashion  irrelevant.  Another  respondent  stated  that  the  information  was 
not  specific  enough:  This  doesn't  indicate  who  will  be  assigned  to  "my"  system— only 

what  the  total  Navy  pool  of  talent  consists  of. 

A  member  of  the  acquisition  community  drew  the  following  conclusions:  The 
designer  sees  his  system  as  getting  the  manpower  because  of  its  "importance;"  it's  the 
other  guy's  system  that  will  be  short  of  manpower.  The  data  given  must  assume  one  set  of 
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Figure  6.  Ratings  of  information  in  Section  4:  Projected  Supply 
of  Technical  Ratings  at  Different  Experience  Levels. 
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conditions;  if  reality  changes,  the  projected  data  will  be  useless.  (On  the  other  hand, 
projections  through  only  1984  aren't  much  good;  a  system  started  now  won’t  be  in  the 
water  by  then.) 


One  senior,  highly-experienced  program  manager  from  the  design  community 
said:  The  Navy  should  place  specific  manpower  requirements  as  part  of  the  System 
Performance  Specification;  I  doubt  if  a  full-range  manpower  study  supported  by  Human 
Engineering  Studies  is  typically  in  order. 

2.  Useful  for  Serious  Manpower  Constraints?  (9) 

Several  comments  echoed  sentiments  expressed  regarding  Useful  for  Present 
Job?  Negative  comments  included:  Of  little  value.  Basically  this  section  is  an  overkill; 
could  be  simplified. 

A  number  of  responses  focused  once  more  on  the  issue  of  who  is  responsible  for 
manpower  availability:  DoD  must  be  responsible  for  specifying  certain  design  constraints 
relative  to  manpower  availability.  The  designer  has  no  control  over  personnel  availability 
but  can  design  to  meet  availability;  however,  he  needs  to  be  told  via  the  specification 
what  the  technical  level  is  to  which  he  should  design;  DoD  should  use  this  information  to 
determine  how  many  personnel  may  be  assigned  to  a  particular  item.  DoD  should  concern 
itself  with  better  technical  utilization  and  organizational  structure  of  shipboard 
maintenance  personnel. 

3.  Accurate?  (4) 

Two  comments  expressed  doubt  concerning  the  accuracy  of  the  data:  Too  many 
variables  influence  manpower  availability.  Data  are  based  on  projections  whose  veracity 
changes  with  assumptions/time.  A  third  comment  suggested  that  although  the  data  looked 
reasonable,  the  commentator  was  somewhat  skeptical  of  long-term  forecasts:  I  would  like 
to  see  tolerances  (limits  of  uncertainty)  on  available  resources  and  needed  resources. 

In  counterpoint  to  the  problems  inherent  in  projecting  availability,  one  respon¬ 
dent  suggested  that  the  projections  did  not  extend  far  enough  into  the  future:  Since  the 
design  process  now  requires  8  to  10  years  from  initial  development  through  first  shipboard 
operational  system,  manpower  availability  must  be  projected  much  further  out  into  time 
to  be  of  any  use  during  the  design. 

4.  Complete?  (6) 

One  comment  on  completeness  echoed  the  desire  for  a  longer-range  projection: 
We  design  systems  now  for  fleet  use  in  1985-1995. 

Two  comments  suggested  that  additional  data  are  needed  to  help  with  interpre¬ 
tation  of  the  projections:  Should  delineate  contingency  variables.  This  type  of 

information  is  especially  apt  to  be  misunderstood;  strongly  urge  that  raw  data  and 
interpretation  be  caveated;  interpretation  of  the  meaning  of  the  data  can  only  be  properly 
made  if  the  history  of  each  rating  is  known;  vast  differences  exist  between  ratings  and, 
therefore,  users  should  be  urged  to  get  additional  data  from  detailers  and  other  sources 
before  they  make  design  decisions. 

One  commentator  cut  to  the  heart  of  the  problem  in  a  particularly  insightful 
interpretation  of  the  availability  data:  As  the  quality  of  personnel  decreases,  what  will 
happen  to  the  skill  requirements?  Promotion  will  continue  but  what  about  skill? 


5.  Understandable?  (2) 


One  commentator  liked  the  graphical  data  but  expressed  the  following  caution: 
Simplistic  thinking  must  be  avoided  in  using  these  data- -suggest  that  a  set  of  review 
criteria  for  interpreting  the  data  and  a  list  of  cautions  be  applied  before  data  are  used  in 
making  design  decisions. 

One  respondent  identified  several  faults  in  the  presentation  which  made  it 
difficult  to  understand:  Format  on  Figures  8,  11,  and  13  show  a  poor  choice  in  the 
vertical  scale;  the  broken  scale  is  misleading  and  not  consistent  with  normal  practice. 
"Numbers  of  men,"  besides  being  overlapping,  is  misleading.  Also,  the  use  of  percentages 
for  available  and  shortfall  is  in  need  of  normalizing  (i.e.,  using  percentages  to  compare 
between  rates  is  meaningless,  so  you  must  rely  on  the  numbers  on  the  axis  in  order  to 
understand  it  and  make  a  comparison). 

6.  Should  be  Changed?  (12) 

Two  commentators  stated  that  this  section  should  be  omitted.  One  of  these 
remarked:  The  projected  supply  problem  is  the  Navy's  responsibility,  not  the  designer's; 
the  Navy  must  start  to  be  responsible  to  the  needs  of  the  designer. 

One  commentator  said  that  contingencies  such  as  national  emergencies,  the 
draft,  etc.,  should  be  presented  and  discussed. 

Three  respondents  commented  that  the  time  frame  of  the  projection  should  be 
expanded.  One  said:  A  20-year  projection  should  be  made  in  order  to  be  consistent  with 
the  expected  life  of  new  systems. 

In  contrast,  another  evaluator  said:  The  data  become  obsolete  very  quickly  and 
are  only  good  for  a  year  or  so  after  they  are  issued;  they  should  not  be  used  for  any  hard 
planning. 


One  commentator  stated  that  the  data  should  be  presented  in  separate  sections, 
one  section  for  each  Navy  rating. 

Comments  from  Other  Sources 


Many  of  the  comments  from  other  sources  echoed  those  that  were  made  in  response 
to  Evaluation  Packages  C  and  G.  Positive  comments  included:  This  is  very  useful 
material;  the  format  is  very  clear.  Manpower  availability  projection  information  is 
generally  useful;  however,  other  sources  than  the  NPRDC  report  should  be  given. 

One  set  of  commentators  from  the  acquisition  community  echoed  a  sentiment 
expressed  by  a  member  of  the  design  community  concerning  the  need  for  the  Navy  to 
exercise  some  responsibility  in  this  area:  Acquisition  people  normally  don't  worry  about 
personnel  supply— maybe  these  graphs  will  bring  attention  to  the  problem. 

Several  commentators  expressed  concern  over  the  variables  that  affect  supply 
projections  and  the  possibility  that  the  data  are  in  error  now  or  will  soon  become  obsolete. 
At  the  very  least,  it  was  suggested,  some  aid  and  direction  should  be  provided  to  help 
users  properly  interpret  the  data. 
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Section  5:  Evaluation  of  Alternatives 


Most  respondents  indicated  that  Section  5  was  both  useful  and  understandable,  with 
the  bulk  of  the  ratings  at  the  higher  ends  of  these  scales.  However,  ratings  were 
distributed  widely  across  the  scales  for  levels  of  accuracy  and  completeness.  The  primary 
concerns  of  respondents  were  that  the  approach  for  evaluating  alternatives  would  not 
work,  the  approach  was  not  relevant  to  design  requirements,  the  methodology  was  too 
subjective,  the  data  provided  were  obtained  from  a  sample  of  judges  that  was  too  small 
and  probably  biased,  and  a  variety  of  improvements  in  the  method  should  be  made. 

Comments  from  other  sources  focused  on  the  utility  of  the  data  to  the  acquisition 
community.  One  subset  of  comments  suggested  that  the  information  was  relevant  to 
system  designers  but  not  acquisition  people.  Another  subset  stated  that  the  information 
was  more  useful  to  the  Personnel  and  Training  Analysis  Office  (PATAO)  within  NAVSEA 
than  to  the  designer  community.  Several  other  comments  pointed  out  problems  in  the 
approach. 

Data  from  Six  Questions  in  Evaluation  Packages 

Figure  7  shows  how  respondents  answered  questions  regarding  the  information  in 
Section  5.  Comments  relative  to  these  questions  are  provided  below. 

1.  Useful  for  Present  Job?  (10) 

Positive  comments  included:  Interesting  approach.  Information  appears  useful 
in  evaluating  designs  (i.e.,  comparing  one  against  another,  but  of  less  use  in  performing 
the  design  function). 

Some  negative  comments  focused  on  the  irrelevancy  of  this  approach  to  current 
design  requirements:  Not  related  to  contract  award  or  requirements.  Waste  of  time;  you 
do  not  have  the  degree  of  latitude  (suggested  here)  but  work  the  tradeoffs  only  within 
Navy-defined  constraints. 

Another  set  of  negative  comments  focused  on  concern  that  the  method  for 
evaluating  alternatives  will  not  work:  Useful  only  if  the  impact  indexes  are  real.  Impact 
indices  are  so  subjective  and  the  comparisons  of  the  21  design  concepts  so  diverse,  I  don't 
think  that  the  summation  of  value  has  any  meaning.  Quantitative  approach  is  good,  but  I 
have  the  impression  that  two  people  with  similar  backgrounds  would  arrive  at  completely 
different  answers  (in  making  up  the  judgmental  data  base).  Even  after  completing  all 
worksheets,  it  appears  unlikely  that  a  clearcut  decision  would  be  indicated. 

One  respondent  asked  the  question  that  must  occur  to  anyone  who  would  like  to 
proceed  to  an  overall  single  index  for  comparing  two  or  more  systems  (as  suggested  at  the 
end  of  Section  5).  This  commentator  asked:  How  can  one  arrive  at  an  impact  index 
weighting  system? 

2.  Useful  for  Serious  Manpower  Constraints?  (12) 

Several  comments  on  this  question  echoed  those  on  the  first  question  regarding 
usefulness.  One  commentator  stated  that  it  would  be  useful  if  imposed  like  M1L-STD-217 
on  reliability. 

Most  comments  suggested  problems  in  the  approach:  Some  sort  of  evalua¬ 
tion/tradeoff  would  have  to  be  done.  It  would  create  more  problems  than  it  solved.  This 
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Figure  7.  Ratings  of  information  in  Section  5:  Evaluation 
of  AT ternatives. 
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is  a  statistically-oriented  procedure  that  does  not  yield  to  the  design  insight  and 
understanding  that  was  achieved  in  Sections  1  and  2  of  the  guide.  Overall,  it  seems  too 
easy;  when  proposals  are  written,  each  contractor  "solves  all  the  problems;"  however,  it  is 
difficult  to  see  how  it  would  be  implemented  and  later  evaluated. 

One  commentator  said:  The  approach  is  based  upon  invalid  assumptions  and  on 
an  idealistic  view  of  the  system  design  process;  many  of  the  areas  listed  are  either 
dictated  or  are  too  undefined  for  evaluation. 

One  commentator  offered  this  analysis:  The  problem  with  this  section  is  "What 
do  the  values  and  summations  mean?"  How  do  you  compare  a  value  of  +118  units  of 
operational  effectiveness  against  -109  units  of  initial  cost?  Values  given  are  some  mix  of 
nominal,  ordinal,  linear,  exponential,  etc.,  scales  without  regard  for  comparability  of 
units.  The  process  would  be  fine  if  it  could  be  shown  to  have  an  interpretable,  valid 
meaning;  regretfully,  it  doesn't  appear  to  have  these  required  qualities. 

3.  Accurate?  (8) 

Most  of  the  comments  under  this  question  focus  on  the  credibility  of  a 
methodology  based  on  subjective  data:  Subject  to  judgment;  not  as  clear  as  shown.  The 
weighting  factors  are  subjective. 

One  group  of  respondents  questioned  the  use  of  subjective  data  and  added  a 
caution  against  using  data  from  a  particularly  small  sample  group:  If  your  sample  group  is 
small  and  biased  your  input  data  are  bad;  hence,  "garbage-in,  garbage-out."  1  have  to  be  a 
little  skeptical  because  of  the  limited  number  of  raters;  were  they  balanced  with  regard 
to  their  respective  backgrounds  and  professional  specialties?  Information  is  only  as  good 
as  the  subjective  evaluations  in  Sections  1  and  2;  given  another  group  of  evaluators 
providing  data  for  Sections  1  and  2,  I  doubt  that  the  statistical  results  would  be  the  same; 
therefore,  all  the  results  of  Section  5  would  deviate  as  well.  This  compounds  fuzzy  data 
in  Sections  1  and  2  with  fuzzy  system  design. 

One  comment  from  a  designer  included  a  lengthy  discussion  and  analysis  of 
several  problems:  Summary  score  compresses  the  variance  contained  in  those  items  which 
were  presented  as  having  substantial  disagreement  among  the  judges.  Perhaps  some  1- 
sigma  or  3-sigma  deviations  should  be  factored  into  the  design  evaluation  summary  to 
reflect  the  substantial  disagreement  among  judges  on  certain  items.  Understanding  the 
role  of  "substantial  disagreement"  items,  and  the  variation  that  they  would  contribute  to 
the  totals  in  Worksheet  5,  may  well  change  the  result  of  the  sample  system  features  that 
were  postulated  and  may  well  change  the  iterative  process  of  tradeoff  following  initial 
analysis.  I  would  strongly  suggest  early  review  of  these  issues  with  the  Navy  Program 
Manager  for  a  particular  system  development;  his  decisions  and  guidance  could  narrow 
down  the  variances  and  tailor  this  tradeoff  analysis  to  his  particular  needs  and  require¬ 
ments. 

4.  Complete?  (7) 

Positive  comments  included:  The  tools  are  useful  in  evaluation,  but  help  is 
needed  in  trading  off  MTBF  and  MTTR  versus  system  operability  and  operational 
capabilities.  The  idea  behind  this  approach  has  merit,  but  it  would  be  helpful  to  point  out 
explicitly  that  the  numerical  results  are  indicators  based  on  the  accuracy  of  the  assigned 
numbers;  if  the  assigned  numbers  have  a  wide  spread  of  values,  then  the  results  will  have 
a  wide  spread;  the  danger  of  trying  to  use  the  numbers  to  obtain  too-fine  results  should  be 
addressed. 
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The  need  for  assigning  weights  for  the  system  design  criteria  was  stated:  Unless 
values  for  weighting  the  14  criteria  are  provided,  the  designer  has  no  objective  basis  for  a 
choice.  More  emphasis  should  be  placed  on  unequal  weighting;  costs  often  dominate 
design  decisions,  sometimes  to  the  exclusion  of  almost  all  other  factors. 

One  commentator  stated:  Not  accurate  for  a  thorough  tradeoff  analysis.  One 
asked:  How  do  you  use  the  system  design  and  evaluation  summary? 

One  respondent  suggested:  An  example  should  be  given  to  show  how  the  impact 
index  is  obtained  from  the  interaction  of  21  design  concepts  and  14  system  design  criteria. 
This  would  give  a  better  understanding  of  the  total  evaluation.  ’ 

5.  Understandable?  (2) 

Both  comments  on  this  question  suggested  some  problems  in  understanding  the 
procedure:  Need  examples  of  two  hypothetical  or  actual  systems  to  show  in  detail  how 
the  impact  indices  are  applied.  1  think  Sheets  5-2  and  5-3  could  be  written  better. 

6.  Change?  (15) 

Positive  comments  included:  Keep  it  like  it  is  but  substantiate  the  numbers. 
Modify  it  for  considerations  such  as  life-cycle  cost  and  recurring  versus  nonrecurring  cost 
tradeoffs  relative  to  hardware  versus  software. 

Negative  comments  generally  focused  on  suspicions  about  the  validity  and 
reliability  of  the  ratings:  This  assumes  demonstrated  validity  and  reliability  of  the 
ratings  which  are  at  the  heart  of  this  technique.  I  won't  argue  with  the  weightings  but  I 
would  maintain  that  strictly  judgment  or  intuition  can  provide  a  wide  range  of  results. 
Too  broad  and  open  to  questions.  A  larger  data  base  would  help  establish  more  confidence 
in  the  Impact  Indices. 

Two  comments  suggested  that  the  procedure  may  be  irrelevant  to  system  design: 
This  kind  of  evaluation,  as  presented,  may  turn  out  to  be  a  paperwork  exercise  that  does 
not  really  contribute  to  design  decisions.  It  always  seems  overly  simple  when  one  adds  up 
a  number  of  measures  to  find  an  answer  to  best  system  design. 

Two  comments  suggested  certain  improvements:  guidelines  should  be  presented 
which  show  how  to  weight  and  translate  the  system  design  and  evaluation  summary;  also, 
impact  index  needs  further  explanation.  Weighting  factors  or  guidelines  for  establishing 
such  factors  would  be  a  useful  addition. 

Comments  from  Other  Sources 

One  group  of  comments  concern  the  utility  of  Section  5  of  the  guide  to  the 
acquisition  community.  One  commentator  said:  NAVELEX  acquisition  people  would  not 
use  this  material.  Another  commentator  stated:  This  stuff  is  only  useful  for  the  actual 
equipment  designer;  NAVSEA  doesn't  do  this  level  of  design. 

On  the  other  hand,  one  evaluator  stated  that  the  Personnel  and  Training  Analysis 
Office  (PATAO)  within  NAVSEA  would  use  this  information:  This  is  the  heart  of  the 
report  for  PATAO  analysts.  However,  you  may  need  to  beef  up  this  section  to  discuss  just 
how  the  evaluation  forms  would  be  applied  in  the  real  world.  I  can  see  application  to  the 
sole  source  evaluation  board  process  for  manpower  and  training  requirements  of  a  new 
system.  System  designers  or  program  managers,  even  if  they  had  the  time  to  try  to  apply 
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this  methodology,  could  do  more  harm  than  good.  It  should  be  used  by  experienced 
PATAO  analysts  trained  in  the  full  implications  of  the  methodology  and  could  be  used  at 
various  phases  of  the  system  design  and  development  process. 

One  commentator  said:  Section  5  purports  to  provide  an  objective  means  to 
determine  "which  general  design  alternative  best  satisfies  not  only  manpower  and  training 
criteria  but  costs,  potential  benefit,  and  technical  risk  considerations."  This  is  an 
overstatement  of  what  is  actually  accomplished  by  the  method  described  in  this  section. 

One  commentator  said:  Section  5  implies  an  analysis  of  two  or  more  systems.  A 
program  manager,  responsible  for  the  delivery  of  a  military  system  within  restricted 
resources,  has  enough  trouble  designing  and  analyzing  one  system.  There  is  not  sufficient 
energy,  time,  or  money  to  devote  to  designing  and  analyzing  two  or  more  systems. 

One  commentator  used  the  guide  to  compare  an  old  flight  deck  communication 
system,  the  AN/SRC-22,  with  a  new  one  that  is  currently  under  development.  While  the 
new  one  had  been  demonstrated  to  be  a  better  system  than  the  old  on  the  majority  of 
criteria  applied  in  a  formal  review,  the  analysis  using  the  guide  showed  that  the  new 
system  was  only  better  in  a  small  number  of  areas  than  the  old.  The  respondent  concluded 
that  the  impact  indices  in  the  guide  are  not  applicable  across  the  board  to  all  systems: 
For  different  systems,  ships,  technology,  etc.,  the  indices  will  be  very  different.  The 
design  concepts  in  the  guide  are  presented  in  a  vacuum.  The  total  tradeoff  problem 
actually  includes  a  larger  number  of  variables  than  considered  in  the  guide.  For  example, 
the  number  of  systems  or  units  to  be  produced  is  one  of  many  variables  that  are  not 
discussed. 

One  very  senior  and  highly  experienced  system  designer  made  the  following  observa¬ 
tion  on  how  the  tradeoff  analyses  in  Section  5  may  be  misleading  in  the  system  design  and 
acquisition  process:  The  sections  where  tradeoffs  on  human  resources  are  made  "have  the 
potential  for  encouraging  contractors  to  'bull'  each  other  and  the  Navy"  because 
everybody  is  going  to  make  it  appear  that  their  proposal  solves  all  problems  and  meets  all 
requirements.  "There  hasn't  been  a  system  proposed  yet  that  can't  be  operated  and 
maintained  by  untrained  Neanderthal  monkeys." 

Section  6:  Taxonomies  of  Tasks  and  Associated  Skill  Levels 


The  pattern  of  ratings  for  Section  6  was  almost  identical  to  those  for  Section  5. 
Although  the  material  was  considered  to  be  useful  and  understandable  by  most  respon¬ 
dents,  its  application  was  viewed  with  a  substantial  amount  of  skepticism.  Comments 
emphasized  the  difficulty  of  using  the  task  taxonomies  and  associated  skill  levels, 
particularly  in  making  the  transition  from  information  based  on  older  systems  to  the 
design  of  new  systems.  The  format  for  presenting  the  information  was  also  criticized. 

Comments  from  other  sources  reiterated  those  from  Evaluation  Packages  C  and  G, 
although  a  wide  variety  of  additional  insights  and  suggestions  was  provided.  No  strong 
common  themes  were  evident  in  the  additional  material. 
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Data  from  Six  Questions  in  Evaluation  Packages 

Figure  8  shows  how  respondents  answered  questions  regarding  the  information  in 
Section  6.  Comments  relative  to  these  questions  are  provided  below. 

1.  Useful  for  Present  Job?  (13) 

Several  comments  focused  on  the  difficulty  in  using  this  material:  The 
information  appears  good  but  the  material  is  hard  to  use.  If  data  were  broken  down  by 
equipment  nomenclature,  it  would  be  more  useful.  Unlikely  that  any  of  this  information 
would  be  used  during  the  initial  stages  of  system  design.  This  doesn't  help  much  except  to 
say  that  the  third  class  petty  officer  needs  close  supervision;  the  percent  time  consuming 
data  doesn't  help  because  relative  times  aren't  given;  not  sure  how  to  use  this  information. 
Not  related  to  contract  award  or  requirement. 

One  respondent  found  the  material  was  too  superficial:  We  developed  the 
Operator  Task  Analysis  and  inputs  to  the  ship's  manning  document;  our  current  experience 
goes  far  beyond  the  material  in  Section  6.  On  the  other  hand,  another  commentator 
suggested  that  the  material  was  too  detailed:  1  cannot  realistically  see  myself  designing 
operator/maintainer  functions  on  a  new  program  which  considers  the  differences  between 
the  first  class,  second  class,  and  third  class  petty  officers;  perhaps  this  might  be  a  fine 
tuning  point  for  upgrading  older  systems. 

Three  comments  focused  on  the  difficulty  in  making  the  transition  from 
information  on  older  systems  to  newer  systems:  Not  very  useful;  for  example,  my  systems 
have  only  a  few  tasks  that  are  common  with  the  SQS-23/26;  I  would  have  to  spend  an 
inordinate  amount  of  time  in  examining  the  23/26  to  find  comparable  tasks.  Tasks  that 
are  identified  in  the  guide  are  too  closely  related  to  specific  systems;  general  categories 
would  be  more  useful.  How  operators  and  maintainers  are  perceived  to  perform  (by  what  I 
believe  is  a  poor  data  sampling  method)  on  old  systems  is  not  germane  to  expected  tasks 
on  new  systems. 

One  evaluator  suggested  that  the  information  is  irrelevant  to  the  way  some  types 
of  design  are  done:  The  equipment  that  1  am  concerned  with  does  not  take  into  account 
any  specific  rating,  but  assumes  a  lower  class  technician  will  most  likely  be  using  and 
maintaining  the  equipment. 

One  commentator  drew  some  conclusions  concerning  design:  An  analysis  of  the 
data  presented  in  this  section  clearly  shows  a  lower  proficiency  in  troubleshooting  and 
replacing  components  (resistors,  capacitors,  etc.).  Newer  systems  which  employ  micro¬ 
miniature  components  would  create  an  even  greater  challenge  to  skill  proficiency  in 
troubleshooting  and  replacement.  These  data  suggest  that  we  eliminate  consideration  of 
onboard  repair  of  LRUs. 

2.  Useful  for  Serious  Manpower  Constraints?  (5) 

One  respondent  found  the  usefulness  of  the  information  lay  in  the  fact  that  it 
developed  insight:  Information  is  useful  to  the  extent  that  I  now  know  that  1  need 
selection  and  training  to  the  second  class  petty  officer  level  before  I  can  get  enough 
talent  to  keep  an  equipment  working.  Third  class  petty  officers  are  obviously  only 
apprentices  and  second  class  petty  officers  are  journeymen. 

Two  comments  suggested  problems  in  using  this  information  to  constrain 
manpower:  I  don't  see  how  these  data  can  be  useful  as  anything  except  a  reference  or 
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Figure  8.  Ratings  of  information  in  Section  6:  Taxonomies  of 
Tasks  and  Associated  Skill  Levels. 


56 


r 


checklist;  the  system  specifications  should  call  out  the  features  that  are  desired. 
Maintainer/operator  tasks  are  defined,  but  to  come  up  with  a  time  consuming  rating  would 
be  hard  to  specify. 

One  commentator  suggested  that  the  material  is  unnecessary:  Designers  with 
any  experience  currently  design  for  least  skill  level. 

One  commentator  deduced  that,  if  this  information  were  imposed  as  a  design 
criterion,  manpower  constraints  would  take  first  priority  and  operator  functions  would  be 
eliminated. 

3.  Accurate?  (8) 

Several  comments  questioned  the  accuracy  of  the  data:  Too  many  limited 
opinions  to  enable  a  statistically  valid  acceptance  of  the  illustrated  data.  I  wonder  if  the 
senior  chief  petty  officers  aren't  conditioned  to  believe  that  rank  and  skill  level  are 
positively  correlated;  the  separations  in  assessed  capabilities  seems  too  uniform.  The 
reader's  confidence  is  diluted  by  the  caveat  and  weak  language  used  on  Page  6-5  of  the 
guide  ("Unanimity  of  opinion  on  this  point  was  unlikely...",  "...relatively  strong  agreement 
that  some..."). 

One  commentator  who  marked  the  "Don't  Know"  category  asked:  Are  all  tasks 
defined?  Are  the  time  consuming  factors  accurate  as  seen  from  the  technician's 
viewpoint? 

4.  Complete?  (5) 

The  few  comments  under  this  question  mostly  echoed  comments  made  under  the 
Useful  and  Accurate  categories.  One  commentator  said:  Would  like  to  see  more  detail 
regarding  the  actual  basic  skill  level  of  each  rating. 

Another  comment  suggested  that  the  breakdown  of  material  by  rating  was  not 
going  to  be  viable  for  very  long:  For  example,  DSs  are  being  phased  out. 

5.  Understandable?  (6) 

One  respondent  stated:  I  like  your  clear  definitions  of  "limited,  partial, 
competent,  superior." 

Negative  comments  included:  Not  easy  to  understand.  The  three  curves  always 
follow  each  other— using  a  shaded  band  or  cross-hatching  might  be  easier  to  read.  I  find  it 
difficult  to  understand  and  work  with  the  graphical  display  of  the  task  taxonomies;  can 
this  representation  be  done  in  a  different  way  entirely  (e.g.,  bar  charting,  tables,  etc.)? 
The  connecting  lines  seem  to  throw  me  off  and  there  is  some  confusion  in  my  mind  as  to 
what  the  differences  among  the  three  petty  officer  technician  levels  really  are. 

Three  comments  focused  on  the  difficulties  in  using  the  material  in  design: 
There  is  no  handy  way  to  find  the  material  required  and  relate  it  to  a  design  problem.  It 
is  difficult  for  the  designer  to  measure  technician  effectiveness  on  a  new  design  released 
to  the  fleet;  true  competency,  based  on  our  experience,  is  not  as  related  to  skill/pay 
grades  as  it  is  to  personnel  motivation  and  individual  skills.  Not  sure  how  to  use  the 
limited  information  presented  here. 
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6. 


Should  be  Changed?  (18) 


A  wide  variety  of  comments  were  offered  under  this  question.  One  comment 
stated:  This  appears  to  be  good  material  but  the  presentation  was  awkward. 

One  commentator  suggested  that  the  material  does  not  give  the  designer  anv 
direction:  What  class  petty  officer  should  be  used  for  what  task?  Does  this  vary  between 
equipments?  What  are  the  minimum  requirements  by  class  for  certain  tasks? 

One  respondent  came  up  with  some  answers  to  the  above  questions:  Consider  j 
summation  of  the  essential  task  difficulty  differences  for  each  petty  officer  level  at  the 
end  of  Section  6;  it  would  be  useful  to  profile  the  differences  according  to  the  following 
generally  understood  technical  levels— apprentice  (P03),  journeyman  (P02),  and  full v 
qualified  technician  (POl). 

Numerous  comments  focused  on  the  difficulty  of  using  information  regarding  ole 
equipments  to  make  design  decisions  on  new  equipments:  This  section's  value  depends  or 
a  new  system  having  tasks  which  are  similar  to  existing  Navy  systems:  At  best,  this 
section  would  serve  as  a  general  guide  for  estimating  operation  and  maintenance  tasks. 
Only  applicable  to  existing  systems;  new  designs  will  be  significantly  different.  Obsolete 
equipment;  this  approach  does  not  consider  the  new  trends  in  integration;  would  like  to 
see  analysis  on  new  equipment  (e.g.,  UYK-43  and  44  replaces  UYK-47  and  20).  The 
problem  is  the  new  type  of  integrated  system,  not  isolated  equipments  or  chassis. 
Requires  definitions  of  example  equipments;  also,  I  could  not  understand  the  func  tion 
performed  by  many  of  the  tasks  that  were  described. 

Several  respondents  focused  on  the  format  of  the  presentation:  I  don’t  like  your 
format,  but  I  don't  have  a  constructive  suggestion.  Figure  15  needs  much  heavier  outlines 
around  the  referenced  blocks.  Explanations  should  be  made  much  clearer  on  the  chart 
headings;  they  should  be  as  self-explanatory  as  possible;  the  current  format  requires  that 
the  reader  refer  back  to  Pages  6-4  and  6-5.  A  different  graphical  representation  of  the 
data  would  be  good  here;  also  the  text  should  be  expanded  to  give  the  user  helpful  hints  as 
to  how  he  can  use  the  data  on  existing  systems  as  points  of  departure  for  determining  the 
manpower  and  training  impacts  on  his  new  design. 

One  senior  engineer  suggested  the  use  of  more  common  language  in  presenting 
the  information  in  Section  6:  After  consulting  my  secretary's  dictionary,  1  found  out  what 
"taxonomy"  means!  You  must  remember  that  most  system  design  engineers  are  just  a 
little  illiterate  (even  us  managers).  We  tend  to  like  simple,  logical  communications. 
Webster's  definition  of  taxonomy  as  an  "orderly  classification"  isn't  too  hard  to  under¬ 
stand.  Why  don't  you  use  those  kinds  of  words.  Try  speaking  our  language  if  you  expect  to 
"sell"  this  guidebook. 

One  comment  suggested  that  a  better  transition  at  the  end  of  Section  6  would  be 
helpful  in  leading  into  Section  7:  Suggest  your  text  in  Section  6  provide  a  bridge  or 
forward  reference  to  Section  7. 

Comments  from  Other  Sources. 

Many  comments  from  the  other  sources  were  similar  to  those  of  respondents  to 
Evaluation  Packages  C  and  G.  The  following  present  some  additional  insights:  1  didn't 
like  the  time-consuming  percentages.  It  was  hard  to  relate  this  statistic  to  what  the 
technicians  are  doing.  I  would  rather  have  seen  the  percentage  of  a  technican's  total  work 
time  that  is  spent  on  each  activity. 
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One  commentator  thought  that  the  guide  should  explain  the  reasons  lor  the  time- 
consuming  tasks:  Why  is  something  time-consuming?  Follow  this  lead  and  provide  the 
information  to  the  designer;  he  won't  dig  it  out  himself. 

One  respondent  noted  the  need  for  additional  data  in  order  to  properly  interpret  the 
time-consuming  percentages:  We  also  need  data  on  the  frequency  of  the  tasks;  are  they 
quarterly,  yearly,  etc.? 

Noting  the  difficulty  in  translating  information  on  tasks  from  old  systems  to  the 
desgn  of  new,  a  group  of  commentators  from  NAVELEX  said:  The  tasks  which  talk  about 
old  style  maintenance  reduce  the  credibility  of  this  document.  Designers  don't  design 
systems  that  way  anymore.  It  might  help  if  you  noted  that  some  of  the  tasks  which  are 
extreme  examples  of  the  above  are  not  currently  accepted  philosophy.  These  com¬ 
mentators  also  admonished:  Reorganize  this  information  by  systems,  not  by  ratings. 

One  individual  made  several  detailed  comments  on  the  material  in  Section  6:  In  order 
for  the  engineer  to  design  a  new  system  that  avoids  tasks  which  involve  difficulty,  he  will 
need  to  find  out  what  features  produced  problems  in  the  old  system.  It  is  not  the  general 
mode  of  operation  for  the  engineer  to  spend  a  great  deal  of  time  examining  old  systems; 
one  doesn't  seem  to  make  much  forward  progress  if  you  keep  looking  backward.  Also, 
instead  of  utilizing  questionnaire  data,  as  in  the  guide,  there  should  be  more  objective  3M 
data  (with  all  its  shortcomings)  that  could  be  used  to  support  analyses  of  "time  to  repair." 
Finally,  the  manpower  and  training  analysis  community  may  be  able  to  use  the  difficulty 
data  to  modify  existing  training  courses,  even  if  the  data  do  not  turn  out  to  be  used  by  the 
design  community. 

Section  7:  Difficult  and  Time-consuming  Tasks 

The  material  in  Section  7  is  essentially  the  same  as  that  in  Section  6,  only  organized 
in  a  different  manner.  Therefore,  it  was  not  surprising  that  the  ratings  and  comments 
were  quite  similar  to  those  obtained  for  Section  6.  One  difference,  however,  is  the 
greater  regard  by  respondents  for  the  accuracy  of  Section  7.  Almost  twice  as  many  rated 
the  accuracy  of  Section  7  in  the  top  two  positions  on  the  scale  than  rated  Section  6  in  the 
top  two  positions.  Also,  fewer  indicated  that  they  didn’t  know  about  the  accuracy. 
Perhaps  Section  7  was  organized  in  a  more  credible  format. 

Data  from  Six  Questions  in  Evaluation  Packages 

Figure  9  shows  how  respondents  answered  questions  regarding  the  information  in 
Section  7.  Comments  relative  to  these  questions  are  provided  below. 

1.  Useful  for  Present  Job?  (9) 

A  number  of  these  comments  echoed  those  from  Section  6  and  will  not  be 
repeated  here.  One  comment  by  a  designer  was  very  positive:  This  is  an  excellent 
concept~why  not  expand  it  to  include  most  probable  tasks  as  well  as  most  difficult? 
Reliability  analysis  should  be  able  to  predict  the  most  probable  causes  of  repair  tasks 
which  can  then  also  be  addressed  in  the  design. 

One  commentator  observed  that  the  material  did  not  lead  to  objective  con¬ 
clusions:  The  task  descriptions  require  the  reader's  imagination  to  develop  a  feel  for  the 
complexity  of  the  tasks. 

The  following  comment  suggested  limits  for  applying  the  information  to  design: 
I  don't  think  it  does  much  good  to  point  out  areas  of  problem  design  since  you  are  only 
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Figure  9.  Ratings  of  information  in  Section  7:  Difficult 
and  Time-Consuming  Tasks. 


itemizing  a  few  problems  from  among  an  almost  infinite  number  of  problem  designs.  The 
data  here  would  be  most  valuable  in  the  refit  of  equipment  or  for  the  next  generation  of 
the  same  equipment. 

Two  remarks  indicated  that  the  difficult  tasks  may  be  intractable:  Can't  really 
eliminate  any  of  these  generic  tasks.  The  information  is  interesting,  but  many,  if  not 
most,  of  the  difficult  tasks  could  not  easily  be  eliminated  from  digital  system  design,  i.e., 
adjusting  a  magnetic  tape  transport. 

One  commentator  appeared  to  discount  the  data  in  Section  7:  I  disagree;  the 
difficult  tasks  are  localization  and  isolation  of  faults.  The  number  of  "glitches"  rises 
exponentially  as  a  function  of  integration  and  modular  design. 

2.  Useful  for  Serious  Manpower  Constraints?  (4) 

Most  comments  reiterated  previous  comments.  New  thoughts  included:  The 
data  would  be  difficult  to  specify,  difficult  to  establish  definitive  design  criteria.  The 
data  might  help  equipment  designers  but  would  be  useless  to  the  system  designer. 

3.  Accurate?  (5) 

The  comments  mainly  reflected  an  uneasiness  about  the  validity  of  the  judg¬ 
mental  data  base:  I  wonder  if  the  senior  chief  petty  officers  aren't  conditioned  to  believe 
that  rank  and  skill  level  are  positively  correlated.  All  the  data  in  the  manual  need  to  be 
backed  up  in  order  to  be  accepted.  1  assume  that  the  judgmental  data  are  based  on 
experience. 

One  commentator  noted:  Some  of  the  tasks  defined  do  not  appear  to  be  operator 
or  maintainer  tasks  (e.g.,  rewriting/writing  programs,  analyze  test  data). 

4.  Complete?  (1) 

The  comment  here  was  by  a  respondent  who  marked  the  "Don't  Know"  response 
box.  He  said:  Are  all  of  the  tasks  identified?  How  well  are  time-consuming  factors 
established? 

5.  Understandable?  (4) 

One  commentator  said:  Color-coded  pages  were  excellent.  Other  comments 
echoed  those  made  on  Section  6  regarding  the  complexity  of  the  format,  the  confusing 
graphics,  and  a  problem  in  the  organization  of  the  material.  The  latter  sentiment  was 
stated:  The  problem  is  in  the  organization  and  in  being  able  to  find  if  the  desired 
information  exists,  where  it  is,  and  whether  there  are  other  applicable  data  elsewhere  in 
the  document. 

6.  Should  be  Changed?  (10) 

Several  suggestions  for  improvement  were  made:  I  have  the  impression  that  the 
data  are  significant,  but  I  am  unsure  about  the  format.  Reorganize  and  provide  good 
indexes.  If  data  were  broken  down  by  equipment  nomenclature,  it  would  be  more  useful. 
Fix  minor  typographical  errors.  Page  9-4,  Paragraph  1.  Need  to  add  more  information  and 
better  guidelines  on  how  to  assign  the  weighting  factors  in  Table  3.  I  think  that  some 
indication  of  what  specific  types  of  tasks  can  be  performed  by  the  different  ratings  would 
be  better  than  the  currently  presented  data  for  suggesting  to  designers  how  to  design  the 
equipment  for  maintenance. 
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One  commentator  stated:  The  information  should  be  useful  as  a  comparison 
guide;  however,  I  do  not  understand  the  exact  requirements  of  many  of  the  tasks  that 
were  presented  in  this  section.  Therefore,  the  difficulty  and  time  requirement  data  are 
meaningless  to  me  as  far  as  making  design  improvements. 

Another  respondent  said:  Section  7  is  difficult  for  a  designer  to  quantify.  1 
understand  the  intent  of  the  document  but  getting  there  is  another  thing.  How  meaningful 
are  the  subjective  measurements? 

One  comment  stated  simply:  Omit  it. 

Comments  from  Other  Sources 

Once  again,  many  of  the  comments  from  other  sources  were  similar  to  comments 
elicited  by  Evaluation  Packages  C  and  G;  these  are  not  repeated  here.  A  group  of 
commentators  from  the  acquisition  community  (NAVSEA)  stated:  This  is  great  material. 
Perhaps  the  main  list  of  tasks  (Section  6  of  the  guide)  could  be  put  in  an  appendix  or 
second  volume  and  just  put  the  list  of  difficult  and  time-consuming  tasks  in  the  main 
report  or  the  first  volume. 

Another  member  of  the  acquisition  community  said:  The  data  on  difficult  tasks  in 
Section  7  purports  to  answer  the  question  "What  skill  levels  and  number  of  operators  and 
maintenance  personnel  are  required  to  perform  these  tasks?"  This  has  been  historically  a 
very  critical  question  in  system  manning  analysis;  this  section  of  the  guide  contributes 
virtually  nothing  to  more  precisely  answering  the  question.  There  is  little  promise  of 
objectively  determining  the  skills  and  manning  level  of  the  system  from  the  data 
presented  in  this  section.  A  commentator  who  is  at  once  a  member  of  the  acquisition 
community  and  the  Navy  design  community  remarked:  Sections  6  and  7  are  interesting 
and  useful,  but  they  do  not  drive  the  designer  toward  a  goal. 

Section  8:  Training  Requirements  and  NECs 

Section  8  was  not  positively  regarded  by  the  sample  of  respondents.  Most 
respondents  considered  that  the  information  was  of  marginal  utility,  accuracy,  and 
completeness.  Also,  some  respondents  had  difficulty  in  understanding  this  section,  and 
others  felt  they  did  not  know  enough  to  respond. 

Additional  data  were  suggested  for  the  section  to  make  it  more  useful  for  designers 
on  their  present  job  and  to  permit  a  satisfactory  response  to  imposed  manpower 
constraints.  However,  many  said  the  material  was  of  little  or  no  value  to  them. 

Data  from  Six  Questions  in  Evaluation  Packages 

Figure  10  shows  how  respondents  answered  questions  relative  to  the  information  in 
Section  8.  Comments  relative  to  these  questions  are  provided  below. 

I .  Useful  for  Present  Job?  (17) 

Positive  comments  included:  Good  reference  data;  I  will  look  forward  to  future 
updates.  In  25  years,  1  never  saw  a  list  of  courses  before. 

A  large  number  of  comments  indicated  that  the  material  was  of  little  or  no 
value:  It  does  have  some  value,  although  very  little.  Not  related  to  contract  award  or 
requirements.  Cannot  correlate  training  requirements  with  the  list  of  ratings  included  in 
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this  section.  Don't  see  how  information  can  help  me  design  a  system;  how  do  I  use  the 
information?  Knowing  what  has  been  given  without  a  feel  for  the  complexity  of  the 
material  covered  in  the  training  and  the  resulting  proficiency  of  the  students  gives  no 
useful  information  for  design  tradeoffs.  Courses  are  equipment-specific;  how  do  I  know 
the  courses  apply  to  my  system?  Data  are  too  limited  in  detail  to  be  useful.  Knowing 
titles  of  NECs  provides  no  useful  information  to  system  designers.  Information  in 
Section  8  is  incomplete. 

Several  respondents  suggested  what  is  needed  to  make  Section  8  useful:  Need 
the  additional  detail  which  was  mentioned  to  be  under  development  and  not  yet  available; 
a  description  of  how  to  access  a  computerized  data  bank  might  be  useful.  List  of  NECs 
could  impact  design  if  detailed  descriptions  of  the  training  for  each  of  the  ratings  were 
included.  Reference  to  the  NEC  Manual  would  contribute  as  much  or  more  than  the 
information  in  its  present  form. 

One  commentator  reiterated  his  remarks  concerning  the  whole  approach  to 
design  in  response  to  manpower  requirements:  This  is  the-cart-before-the-horse  "bull." 
Training  requirements  and  NECs  should  be  driven  by  design  requirements,  not  the  reverse. 

2.  Useful  for  Serious  Manpower  Constraints?  (II) 

Some  commentators  pointed  out  the  need  for  additional  data  in  order  to  make 
this  information  useful  in  responding  to  serious  manpower  constraints:  Information  on 
NEC  costs  for  various  technical  ratings  is  vitally  needed.  If  this  information  were 
available,  it  would  be  of  great  assistance  to  any  manpower  and  training  support  analyst. 
The  section  is  missing  some  key  data  which  are  under  development;  at  the  present  it 
would  not  be  of  much  use.  Need  facts— how  much  does  training  cost?  How  long  does  it 
have  to  be?  How  successful  is  it?  Availability  of  courses  and  content,  devices  and 
facilities  should  be  summarized  here. 

Two  respondents  saw  problems  in  the  data:  Implicit  in  these  data  are  the 
assumption  that  new  equipment  can  or  should  be  designed  to  utilize  existing  NECs;  this  is 
a  fallacious  assumption!  There  doesn't  seem  to  be  a  significant  correlation  between 
manpower  and  available  courses. 

Two  commentators  rejected  the  use  of  this  type  of  data:  This  is  a  DoD  problem. 
This  would  become  another  requirement  to  be  contended  with  by  the  designers;  it  would 
add  little  to  system  design. 

3.  Accurate?  (3) 

These  comments  again  pointed  out  that  the  section  is  missing  key  data  which  are 
under  development.  One  comment  brought  to  light  a  new  thought  concerning  the 
accuracy  of  these  data:  Courses  like  TALOS  are  listed,  but  the  school  closed  2  years  ago. 

4.  Complete?  (10) 

Many  comments  on  this  question  echoed  points  that  have  already  been  made; 
these  will  not  be  repeated.  One  commentator  noted  the  lack  of  specific  information  that 
should  have  been  included:  the  MK-86  FSC  NEC  should  be  included;  this  training  course 
has  existed  since  1971-72.  One  commentator  noted  the  need  to  make  the  material  up  to 
date:  Doesn’t  list  courses  that  are  about  to  be  made  available. 
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Two  comments  indicated  that  the  data  were  too  general:  The  data  are  in  terms 
of  relative  factors:  need  better,  exact  guidance  to  be  able  to  tradeoff  nonrelative  factors. 
Needs  amplification  of  actual  skills  and  knowledge  possessed  by  each  NEC. 

5.  Understandable?  (4) 

Most  of  the  commentators  echoed  earlier  thoughts.  One  new  idea  was  proposed: 
Listing  is  raw  data;  a  cross-reference  to  task  types  would  be  better. 

6.  Should  be  Changed?  (19) 

Many  of  the  comments  on  this  question  reiterated  previous  sentiments.  The 
following  are  general  suggestions:  Either  drop  it  or  expand  it.  It  needs  to  be  more 
complete;  it  is  currently  of  little  practical  use  to  the  designer;  it  is  general  information 
only.  This  section  doesn't  say  anything;  the  references  are  nice,  though.  Data  needs  to  be 
expanded  and  reformatted.  Does  not  have  data  that  are  of  value  to  the  designer— except 
in  a  general,  informative  manner.  Omit  section  until  data  which  are  referred  to  on  Page 
8-2  become  available.  This  would  probably  best  be  presented  in  an  appendix  since  it  does 
not  add  to  the  process  of  determining  training  requirements. 

The  following  are  some  more  specific  suggestions  for  improving  the  material:  A 
tie-in  to  training  would  be  useful;  present  the  number  and  the  duration  of  classes  to  relate 
to  the  level  of  required  experience.  Provide  some  means  to  evaluate  the  level  of  training 
received  and  the  average  proficiency  of  students  at  the  end  of  formal  training.  There  is 
too  much  redundancy  among  the  many  courses  presented;  the  entire  list  could  be 
condensed  by  skill  category.  Expand  the  section  and  give  a  basic  outline  of  material 
covered  in  each  course.  The  section  should  be  expanded  to  give  the  designer  more  insight 
into  the  problems,  availability,  and  cost  of  training.  The  data  for  life  cycle  costs  involved 
in  developing  new  NECs  should  be  presented.  Provide  an  outline  for  a  typical  NEC  and 
the  cost  factors  involved  in  developing  a  new  NEC. 

Comments  from  Other  Sources 

One  group  of  commentators  from  the  acquisition  communitity  (NAVELEX)  stated: 
This  is  very  good  and  useful  information.  Another  group  of  commentators  from  the 
acquisition  community  (NAV5EA)  stated:  This  is  not  useful;  the  designer  will  rely  on  the 
training  plans  conference  for  this  kind  of  input. 

Comments  on  the  inadequacy  of  the  material  presented  in  Section  8  included:  It  is 
only  a  pretense  to  think  that  this  information  answers  the  question  that  it  claims  to 
address.  It  is  inadequate  just  to  present  the  number  and  title  of  the  NEC;  why  not  also 
provide  the  NEC  descriptions  from  the  NEC  Manual? 

A  manpower  and  training  analysis  expert  from  the  acquisition  community  suggested: 
This  section  needs  further  expansion  to  be  of  maximum  use.  The  list  of  NECs  is  woefully 
out  of  date.  Give  a  few  examples  of  system  NECs  under  each  rating  and  delete  the 
complete  lists  which  are  currently  provided  in  the  guide  and  which  can  be  more  readily 
obtained  from  a  current  NEC  Manual.  Provide  more  discussions  of  NEC  development  to 
show  the  role  that  NECs  play  in  manpower  and  training  analysis  and,  consequently,  how 
that  may  affect  the  selection  of  design  alternatives.  The  NEC  life  cycle  cost  data,  which 
is  apparently  under  development,  is  vital  to  making  this  section  useful. 
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Section  9;  Billet  Life  Cycle  Costs  for  Required  Personnel 


Respondents  had  little  difficulty  understanding  the  information  in  Section  9,  and  most 
indicated  by  their  ratings  that  the  information  on  billet  life  cycle  costs  was  useful.  Both 
accuracy  and  completeness  of  the  information  were  questioned,  however,  by  a  significant 
proportion  of  respondents.  In  addition,  many  felt  they  did  not  know  enough  to  assess 
accuracy  and  completeness.  Part  of  the  credibility  of  the  section  was  lost  because  of  two 
inconsistencies  between  dollar  amounts  in  the  tables  and  the  text.  Many  comments 
pointed  out  these  errors,  using  the  error  to  justify  low  ratings.  In  addition  to  suggestions 
for  correcting  these  errors,  suggestions  were  made  to  expand  the  information  to  make  it 
more  useful. 

Data  from  Six  Questions  in  Evaluation  Packages 

Figure  11  shows  how  respondents  answered  questions  relative  to  the  information  in 
Section  9.  Comments  relative  to  these  questions  are  provided  below. 

1.  Useful  for  Present  Job?  (8) 

Several  commentators  stated  that  the  information  was  not  useful  to  a  designer's 
current  iob:  This  information  is  not  particularly  useful  to  a  designer.  A  typical  designer 
hasn't  got  time  for  this;  it  may  be  appropriate  for  a  specialized  analyst.  If  I  am  doing  life 
cycle  costs,  this  is  of  general  interest;  otherwise,  it  is  of  little  value  in  designing  systems. 
Life  cycle  cost  is  only  a  secondary  requirement  for  my  present  job  in  systems  engineering. 
The  first  time  I  ever  saw  such  a  table;  these  numbers  have  been  quoted  by  others  though. 

One  respondent  suggested  that  the  Navy  would  not  appreciate  design  inputs 
based  on  this  type  of  information:  Navy  not  receptive  to  life  cycle  costs  from  a  designer's 
standpoint. 

One  commentator  made  the  following  suggestion:  Data  on  specific  life  cycle 
costs  by  rate  are  not  useful.  However,  a  rule  of  thumb  for  a  technician  at  the  E-4  or  E-5 
paygrade  would  be  more  useful.  The  variation  between  ratings  (job  specialties)  is  small 
enough  that  a  rule  of  thumb  would  be  adequate.  These  numbers  are  also  small  compared 
to  system  acquisition  costs  and  system  support  costs  (other  than  manning). 

2.  Useful  for  Serious  Manpower  Constraints?  (5) 

One  comment  indicated  that  this  information  would  not  be  useful  to  the 
designer.  Another  comment  reiterated  the  earlier  sentiment  that  this  was  not  the 
designer's  concern:  Tell  the  Navy  to  work  more  efficiently  with  less  manpower  and 
reduced  payroll. 

A  member  of  the  acquisition  community  foresaw  the  following  problem:  If  DoD 
constrained  a  system  to  be  built  with  a  low  budget  for  manning,  they  might  constrain  the 
system  right  out  of  existence. 

The  following  suggestions  were  made  to  make  the  material  relevant  for  serious 
manpower  constraint:  Should  be  updated  for  each  contract  to  reflect  current  economic 
factors.  The  information  must  be  in  current  fiscal  year  dollars. 

3.  Accurate?  (10) 

A  large  number  of  comments  pointed  to  two  errors  in  this  section.  One  error 
appears  on  Page  9-4  in  the  third  paragraph,  where  the  dollar  amount  at  the  end  of  the 
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Figure  11.  Ratings  of  information  in  Section  9:  Billet 
Life  Cycle  Costs  for  Required  Personnel. 


paragraph  is  for  a  10-year  life  cycle  cost  instead  of  the  1  5-year  life  cycle  cost  called  for. 
The  other  error  occurs  in  the  last  table  on  Page  9-6,  where  dollar  amounts  in  the  example 
problem  are  for  10-year  life  cycle  costs  instead  of  15-year  life  cycle  costs  called  for. 

Two  comments  expressed  concern  over  the  basis  for  computing  life  cycle  costs. 
One  said:  Do  these  costs  include  allowances  for  leave,  sickness,  or  other  time  off?  A 
manpower  and  training  analyst  stated:  Various  cost  models  and  data  exist  within  the 
Navy;  this  is  just  one  of  them;  input  factors  are  always  open  to  debate  and  are  questioned; 
why  not  mention  advantages  and  limitations  of  this  model  and  point  out  other  models 
which  can  be  used  for  budget,  funding,  and  program  justification? 

4.  Completeness?  (4) 

One  commentator  noted  that  a  large  cost  factor  group,  operational  specialists, 
were  missing  from  the  data. 

Two  comments  suggested  the  need  for  more  information  on  technical  matters: 
Need  a  more  detailed  discussion  of  the  10  percent  discount  rate.  What  about  escalation, 
cost  growth,  forward  pricing? 

5.  Understandable?  (1) 

The  comment  was  in  reference  to  the  errors  in  the  text. 

6.  Should  be  Changed?  (8) 

Several  of  the  suggestions  concerned  fixing  the  errors  in  the  text  and  tables. 
Two  commentators  stated  that  the  material  was  irrelevant  to  design:  Omit  the  material; 
it  is  a  Navy  problem.  Section  9  could  be  eliminated;  it  is  of  no  real  interest  to  design 
criteria. 


Several  respondents  suggested  changes  that  would  expand  the  information  to 
make  it  useful:  Text  could  be  expanded  to  discuss  other  cost  factors  which  have  not  been 
included  in  this  presentation.  Delete  Table  4  and  provide  more  examples.  More  backup 
information  is  necessary  in  order  to  modify  the  current  situation.  Modify  this  material 
with  specific  historical  examples  of  Navy  manning  topics— watch  standing,  preventive 
maintenance,  etc.,  for  each  type  of  operator  or  maintainer. 

Comments  from  Other  Sources 

On  the  positive  side:  The  billet  life  cycle  cost  data  presented  in  Section  9  is  a  clear, 
straightforward  presentation  and  is  valuable  reference  material.  The  cost  data  are  very 
desirable;  however,  they  must  be  updated  yearly. 

The  following  suggested  some  questions  as  to  the  relevance  of  the  material  to  design: 
This  is  useful  information  to  higher  level  people,  not  to  designers.  Useful  but  not 
indispensable  information;  how  is  it  used  in  the  work  sheets? 

One  commentator  discussed  the  problems  of  using  life  cycle  costing  considerations  in 
system  design:  Section  9  contains  the  central  elements  required  to  drive  design  if,  and 
only  if,  total  life  cycle  costing  can  be  imposed  in  the  acquisition.  Often  life  cycle  costing 
cannot  be  imposed  because  it  conflicts  with  constraints  in  the  R<5cD  budget.  In  the  near 
term,  other  specifications  constraints  must  be  imposed  to  drive  design  in  the  right 
direction  until  life  cycle  costing  finally  (if  ever)  comes  of  age. 
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Section  l(h  References 


The  references  were  not  considered  to  be  very  useful  or  relevant  by  the  sample  of 
respondents.  Interestingly,  many  comments  indicated  that  the  references  could  only 
become  relevant  if  they  were  made  part  of  a  contractual  document.  Relatively  large 
percentages  claimed  they  were  not  sufficiently  knowledgeable  to  assess  the  references. 

For  example,  nearly  half  did  not  feel  qualified  to  rate  the  completeness  of  Section  10. 
Many  suggestions  were  made  to  expand,  annotate,  categorize,  and  index  the  references  to 
make  them  more  useful. 

Data  from  Six  Questions  in  Evaluation  Packages 

Figure  12  shows  how  respondents  answered  questions  regarding  the  information  in 
Section  10.  Comments  relative  to  these  questions  are  provided  below. 

1.  Useful  for  Present  Job?  (10) 

Most  of  the  comments  on  this  question  suggested  that  the  references  weren’t 
useful  or  relevant:  In  the  normal  work  environment,  there  is  not  time  to  do  much  library 
work.  Why  should  the  designer  have  to  read  all  those  references?  Reference  utility  would 
be  primarily  in  evaluating  this  Manual.  Not  very  helpful  unless  one  wishes  to  specialize  in 
the  subject.  The  material  extracted  from  the  references  is  useful,  but  the  references 
themselves  are  of  no  value  unless  I  am  tasked  with  reading,  analyzing,  and  criticizing 
them.  If  this  guide  is  to  be  the  authority,  it  doesn't  need  references.  They  are  not 
required  for  current  contracts. 

Two  comments  treated  the  problem  in  accessing  the  material  if  the  designer 
wished  to:  References  not  available  to  the  typical  designer.  These  references  are  from  a 
wide  variety  of  sources;  if  I  needed  one  of  them,  our  librarian  might  have  a  tough  time 
finding  a  copy;  I  recommend  that  these  references  be  kept  as  a  single  library  somewhere 
that  all  users  can  access. 

2.  Useful  for  Serious  Manpower  Constraints?  (8) 

Some  remarks  echoed  the  sentiments  expressed  on  the  first  question.  Three 
comments  related  the  utility  of  the  references  to  contractual  requirements:  References 
are  not  very  useful  unless  one  wishes  to  specialize  in  this  subject;  of  course,  if  this 
concept  were  to  be  imposed  contractually,  the  references  would  be  of  value.  References 
might  be  of  value  in  the  event  of  a  serious  DoD  push  to  do  something  constructive.  No 
definitive  requirements  are  stated. 

One  of  the  commentators  believed  that  the  references  were  irrelevant:  None  of 
the  references  truly  addresses  the  problem. 

Two  comments  indicated  that  the  references  should  be  indexed  or  annotated: 
This  list  of  references  is  useful  as  a  beginning  but  needs  to  be  greatly  expanded;  also,  the 
various  types  of  references  should  be  categorized  with  annotations  as  to  their  intended  or 
suggested  use.  I  would  not  know  what  documents  provided  a  particular  set  of  data. 

3.  Accurate?  (4) 

Two  of  the  four  comments  on  this  question  stated  that  the  commentators  could 
not  assess  the  accuracy  of  the  references. 


69 


USEFUL  FOR 
PRESENT  JOB? 


L 


VERY  USEFUL 


COMPLETELY  useless 


2.  USEFUL  FOR 
SERIOUS 
MANPOWER 
CONSTRAINTS? 


VERY  USEFUL 


COMPLETELY  USELESS 


3.  ACCURATE? 


4.  COMPLETE? 


VERY  ACCURATE 


“S’ 

s'  ^ 


VERY  QUESTIONABLE 


_ 1 


THOROUGH  AND  COMPLETE 


full  OF  gaps 

AND  OMISSIONS 


5.  UNDERSTANDABLE? 


L  _.J 


EASY  TO  UNDERSTAND 


-=l 


HARD  TO  UNDERSTAND 


6.  SHOULD  BE  CHANGED? 


'~1 


.J 


KEf  P  AS  IS 


CHANGE  OR  OMIT 


PERCENT 
"DON'T  KNOW" 


Figure  12.  Ratings  of  information  in  Section  10:  References. 
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Two  comments  suggest  that  the  particular  references  in  the  guide  miss  the  mark: 
I  have  read  most  of  the  references  and  I  really  don't  feel  that  the  subject  in  question  is 
addressed  from  the  system  designer's  point  of  view.  Suggest  you  look  for  more  industrial 
contractor  references,  like  the  last  one  on  Page  1-2. 

4.  Complete?  (1) 

The  one  comment  under  this  question  said  that  the  commentator  had  no  idea 
about  the  completeness  of  the  list  of  references. 

5.  Understandable?  (0) 

6.  Should  be  Changed?  (11) 

Two  of  the  comments  on  this  question  indicated  that  the  section  should  be 
dropped:  To  be  useful,  the  guide  should  stand  alone.  A  bibliography  is  of  no  particular 
value  to  the  program  or  system  engineers. 

The  larger  proportion  of  comments  suggested  that  the  reference  material  be 
increased:  Expand  it.  Expand  it  and  index  it.  Change  to  an  annotated  bibliography. 
Could  use  some  abstracts.  Provide  short  extract  of  what  each  report  covers  and  contains 
and  the  validity  base  of  the  reported  data.  Updating  of  new  information/references 
should  be  a  follow-on  project. 

One  commentator  suggested  the  following:  Present  some  kind  of  tabular 
representation  that  categorizes  the  reference  materials  and  shows,  under  each  category, 
what  kind  of  assistance  it  will  provide  in  relating  manpower  and  training  requirements  to 
systems  design.  Also,  provide  a  contact  list  of  DoD  and  Navy  organizations  which  can  be 
a  source  of  information  and  counsel. 

Comments  from  Other  Sources 

A  commentator  from  the  acquisition  community  suggested:  This  section  is  totally 
inadequate.  Should  be  revised  and  greatly  expanded.  Don't  merely  give  a  list  of 
references,  provide  several  pages  of  text  discussion  to  explain  different  types  of 
references,  their  potential  uses,  and  other  sources  of  help  within  Navy  organizations;  the 
latter  could  be  categorized  by  codes  and  functions. 

A  group  of  respondents  from  the  design  community  suggested:  The  guide  should 
include  a  list  of  military  specifications.  What  does  each  specification  cover?  Include 
specifications,  standards,  Data  Item  Descriptions,  etc.  Requisition  people  don't  study 
these  specifications,  they  just  include  them  in  the  contract  package  and  nobody  knows 
which  ones  are  really  relevant. 

The  commentator  from  the  acquisition  community  suggested  the  following  documents 
should  be  referenced  in  the  guide:  New  Development  Human  Factors  Program  Guide, 
BUPER5,  18659A;  Human  Factors  Technique  Employed  in  Deriving  Personnel  Require¬ 
ments  in  Weapons  Systems  Development,  October  1967,  Report  No.  PRR68-3;  Personnel 
and  Training  Research  in  Support  of  Advance  Ship  Programs  (Rev),  Volumes  I  and  II, 
March  1971,  PTB71-2;  Guide  for  Conducting  Personnel  Training  Research  in  Support  of 
Advance  Ship  Programs,  May  1970,  PTB-70-7. 


Comments  on  the  Guide  as  a  Whole 


Many  comments  could  not  be  categorized  and  treated  under  the  specific  sections 
within  the  guide.  These  comments,  which  represented  evaluators'  reactions  to  the  guide, 
its  purpose,  approach,  and  content,  as  a  complete  entity,  provided  some  of  the  most 
insightful  critiques  and  useful  suggestions  regarding  the  guide.  For  facility  in  preparation, 
these  general  comments  have  been  grouped  into  seven  themes:  general  critique,  the  need 
for  a  document  like  the  guide,  utility  of  the  guide,  suggested  users  of  the  guide, 
implementing  the  guide,  suggestions  about  the  content  of  the  guide,  and  suggestions  on 
the  format  of  the  guide.  In  the  following  paragraphs,  we  have  identified  the  approximate 
level  and  type  of  person  who  made  each  comment  to  provide  some  perspective  for 
interpretation.  Also,  it  should  be  noted  that  a  given  comment  under  one  theme  may  also 
pertain  to  another. 

General  Critique 

1.  Contractor  (Senior  Systems  Engineer).  Personally,  1  found  the  book  informative 
and  plan  to  retain  it.  Certainly  a  book  of  this  type  is  needed  and  long  overdue. 

2.  NAVELEX  (Technical  Director  of  a  Directorate).  The  guide  iucidly  and  correctly 
pinpoints  the  issues  involved  and  the  clear  responsibility  of  the  systems  designer  to 
consider  the  human  resources  factor  as  an  element  of  system  design.  Adequate  generic 
coverage  is  provided  of  the  tradeoffs  that  can  and  should  be  made  between  skill  levels  of 
human  resources  versus  complexity  of  electronic  systems  in  the  concept  of  life  cycle 
costs.  The  guide  can  be  a  useful  tool  for  systems  designers,  particularly  our  hardware 
contractor  engineers  who  in  many  cases  have  little,  if  any,  appreciation  of  the  personnel 
resource  and  skill  level  conditions  in  the  Navy. 

3.  Contractor  (Program  Manager).  Some  of  the  sections  were  very  good.  Some 
were  just  so  much  paper.  Section  7  on  Difficult  and  Time-Consuming  Tasks  and  Section  6 
on  All  Tasks  had  a  lot  of  detail  that  was  very  good  and  very  useful.  Those  are  the  kinds  of 
things  that  you  really  have  questions  on.  I  am  impressed  with  the  amount  of  work  that 
went  into  putting  the  guide  together.  This  kind  of  data  are  very  important  to  us.  It's  not 
that  bad  that  the  first  cut  at  making  a  guide  like  this  should  have  some  inaccuracies.  I 
want  to  encourage  the  continuation  of  the  development  of  these  data.  If  the  data  were 
tuned  up,  the  guide  would  have  an  overall  beneficial  impact  on  the  design  process. 

4.  Contractor  (Senior  Systems  Engineer).  My  comments  were  brief  due  to  my  basic 
disagreement  with  the  whole  approach  to  the  book  and  some  of  its  underlying  assumptions. 

5.  Contractor  (Publications  Manager).  I  think  the  design  guide  is  a  fine  idea. 

The  Need  for  a  Document  Like  the  Guide 

1.  NAVELEX  (Technical  Director  of  a  Directorate).  Those  of  us  experienced  in  the 
ship  design  discipline  have  long  realized  that  the  human  being  is  the  most  resource- 
consumptive  item  (from  a  life  cycle  cost  standpoint)  that  must  be  put  aboard  ship.  As  we 
put  more  and  more  complex  systems  aboard  ship  to  improve  and  expand  the  ship's 
capability,  the  controlling  factor  is  not  so  much  the  skill  level  of  each  individual  per  se  as 
it  is  the  numbers  of  personnel  required  at  any  given  skill  level  to  operate  the  ship.  Thus, 
the  incentive  for  automation,  high  reliability,  ease  of  maintenance,  and  self-diagnosis,  are 
tied  directly  to  reducing  manpower  that  must  be  put  aboard  and  increasing  the 
productivity  and  motivation  of  that  manpower,  while  at  the  same  time  increasing  the 
combat  capability  of  the  ship.  It  is  considered  that  the  widespread  circulation  and  use  of 
this  guidebook  will  promote  this  objective. 
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2.  NOSC  (Division  Head).  While  responsibility  for  consideration  of  human  resources 
(in  fact,  all  support  elements)  as  an  element  of  design  rests  with  system  designers,  there 
is  not  now  any  method  of  contractually  requiring  this  action.  The  majority  of  all  system 
design  work  is  done  in  industry;  therefore,  no  requirement  is  ever  translated  to  the 
designer.  Furthermore,  support  requirements  in  general  are  not  established  in  conceptual 
phases  except  in  a  most  cursory  manner;  therefore,  review  points  (NSARC,  DSARC,  etc.) 
have  no  criteria  on  which  to  judge  the  proposed  development. 

Manpower,  personnel  and  training  system  concepts  are  not  routinely  required. 
No  restraints  on  manpower  utilization  or  skill  levels  are  imposed.  Follow-on  1LS  planning 
is  driven  by  design  with  no  attempt  to  reduce  ILS  requirements. 

Acquisition  process  decisions  are  driven  by  the  sponsoring  source  of  funding  (i.e. , 
R&D,  not  LCC  considerations).  LCC  guidelines  are  not  established  for  developmental 
systems. 

Utility  of  the  Guide 

1.  Contractor  (Engineering  Program  Manager).  I  enjoyed  the  evaluation  and  learned 
a  great  deal.  We  could  have  used  this  manual  in  Trident  and  Captor.  We  could  have  used 
it  to  design  the  ground  support  system  for  Captor;  we  ended  up  with  embedded  computers 
that  have  since  turned  out  to  be  a  big  pain.  It's  difficult  to  obtain  and  train  people  to 
maintain  the  sophisticated  electronics.  Would  like  to  keep  the  book.  Would  like  to  see 
data  base  expanded.  Would  like  to  see  the  data  base  updated  regularly.  If  possible,  please 
keep  us  advised  as  to  when  it  may  become  a  document  that  we  can  use  in  our  work. 

2.  OPNAV  (?)■  The  guide  may  be  more  useful  in  comparison  of  candidate  systems  in 
order  to  determine  which  is  the  more  cost-effective  for  purposes  of  awarding  a  contract. 

3.  Contractor  (Chief  Electronics  Engineer).  We  are  very  aware  of  the  maintenance 
problem.  I  understand  exactly  what  you  are  trying  to  do  with  the  guide.  It's  the  Navy 
customer  that  makes  life  tough  for  us;  we  tell  him  that  if  he  wants  all  the  fancy  features 
on  the  system,  it's  going  to  take  a  large  amount  of  training;  however,  if  he  can  live 
without  certain  features  and  requirements,  then  training  can  be  cut  down  accordingly. 
The  guide  will  be  a  useful  document  to  prove  this  point.  In  the  past,  we  have  been  saying 
that  you  can  eliminate  a  maintenance  man  and  save  a  certain  number  of  dollars,  but  there 
has  never  been  any  compelling  proof  for  this  argument.  Now  we  can  use  the  guide  to 
prove  the  argument  and  show,  additionally,  that  the  Navy  can't  provide  the  manpower  any 
more  to  maintain  all  the  fancy  features. 

4.  NOSC  (Division  Head).  The  data  are  an  average  that  doesn't  mean  much. 
Different  ships  and  different  implementations  would  lead  to  different  results.  You  could 
not  plug  this  information  into  a  formula  and  trust  the  result.  It  is  very  good  if  you  take  it 
with  a  grain  of  salt  and  think  about  it,  but  not  as  a  cookbook.  If  the  guide  were 
reformatted,  it  could  be  useful  in  conjunction  with  our  Project  Manager's  Guide. 

5.  Contractor  (Electronics  Engineering  Manager).  I  liked  the  guide  and  think  it  is 
quite  useful  in  many  systems.  I  would  like  a  copy  of  the  final  report. 

6.  NESSEC  (RDT&E,  Staff).  COMSEC  Limited  Maintenance  (board  substitution) 
can  only  be  done  by  technical  personnel  who  have  attended  a  specific  course  for  each 
equipment.  Therefore,  each  ship  must  have  a  maintenance  man  qualified  for  each 
equipment.  The  21  design  concepts  in  Section  1  will  be  useful  in  developing  future 
COMSEC  repair  and  maintenance  concepts. 


7.  NOStJ _ ( 1 1 1  Manager).  The  guide  helps  by  providing  mlorriiation  on  man¬ 

power  availability  and  explaining  what  the  design  concepts  do  and  how  they  intera<  t. 

Suggested  Users  of  the  Guide 

1.  NAVELEX  (Deputy  Commander  for  a  Directorate).  The  guide  has  been  reviewed 
by  several  codes  within  NAVELEX.  Those  who  have  reviewed  the  study  carefully  have 
agreed  that  it  would  be  useful  to  hardware  design  contractors.  At  present  it  is  not  in  a 
form  that  can  be  imposed  on  contractors  as  a  specification,  military  standard,  etc. 
However,  it  can  be  furnished  as  information  to  the  design  contractors.  The  study  would 
not  be  useful  to  contractors  that  are  working  to  a  clearly  defined  production  specifica¬ 
tion.  Only  those  in  the  design  phase  would  find  it  helpful. 

2.  NAVSEA  (1LS  Engineer  in  a  PM  Office).  It  seems  like  the  kind  of  stuff  that  an 
ILS  person  like  myself  could  use.  Program  manager  might  use  it;  APM  would  not. 

3.  Contractor  (Engineering  Program  Manager).  1  found  the  work  very  useful  to 
system  designers  and  managers.  The  surveys  in  Sections  1,  2,  6>,  and  7  were  particularly 
good. 

4.  NAVELEX  (Branch  Head).  The  users  of  the  guide  might  be  the  program  manager, 
technical  director,  and  project  engineer.  People  who  work  on  advanced  development 
models  (ADM)  should  also  use  the  guide;  thus,  the  breadboard  models  would  be  a  lot  closer 
to  what  the  production  models  should  be. 

We  could  have  used  the  guide  in  redesign  of  the  SPN-42  when  we  were  deciding 
on  the  type  of  display  system  to  be  used*  The  guide  would  have  advised  of  the  troubles  of 
using  equipments  which  require  chilled  water  systems. 

The  guide  would  also  be  very  useful  for  new  contractors  who  don't  know  much 
about  the  operating  Navy. 

5.  Contractor  (Systems  Discipline  Engineer).  Perhaps  the  document,  with  some 
revisions,  could  be  useful  as  a  management  evaluation  tool.  However,  1  think  the  entire 
document  is  of  little  use  as  a  design  guide.  Most  of  the  information  presented  is  well 
known  to  any  system  designer  with  any  degree  of  experience.  The  information  that  is 
"new"  may  be  okay  but  it  is  of  little  use  in  evolving  a  design. 

6.  NAVSEA  (Assistant  Program  Manager,  Manpower  and  Training  Analysis).  I  feel 
that  this  report  can  be  immensely  useful  to  a  great  many  people  like  myself  who 
participate  in  system  acquisition.  It  is  a  great  relief  for  once  to  read  a  report  where  the 
methodology  does  not  depend  on  complicated  formulas  and  which  can  be  understood  by  a 
wide  variety  of  potential  users.  The  potential  application  of  the  guide  to  the  Navy's 
Source  Selection  Evaluation  Process,  especially  during  the  Source  Selection  Evaluation 
Board  (SSEB)  proceedings,  alone  would  almost  justify  its  development. 

7.  NAVSEA  (Manpower  and  Training  Analyst).  In  my  opinion,  the  document  is  an 
excellent  tool  and  roadmap  for  hardware  acquisition  managers  and  senior  svstem 
designers.  It  should  be  of  valuable  assistance  to  them  in  assessing  their  manpower 
requirements  and  the  cost  implications  of  alternative  electronic  system  configurations. 
The  approach  and  methodology  reflected  in  the  guide  appear  to  be  sound  and  sufficiently 
broad  enough  in  scope  to  measure  all  major  design/cost  elements.  A  few  points  for 
consideration  are: 
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If  the  document  is  programmed  to  be  the  engineer's  guide  or  bible, 
some  thought  ought  to  be  given  to  the  maintenance  of  the  data  bases 
(e.g.,  questions  of  validity,  methodology,  projections  of  personnel 
availability,  billet  costs,  etc).  I  see  a  maintenance  requirement 
relative  to  the  content  of  the  data  in  the  guide,  updating  procedures, 
and  responsibility  for  doing  all  of  this.  Also,  loose  leaf  binders  might 
be  more  advantageous  from  an  updating  standpoint,  rather  than  a 
permanently  bound  document. 

Projections  of  manpower  availability  might  be  expanded  to  include 
other  ratings  (e.g.,  OTs,  PMs,  CTMs,  DPs,  etc.),  since  design 
considerations  may  include  complex  operator  requirements  and  the 
operator  availability  of  appropriate  junior/senior  pay  grades  may  not 
support  the  implementation  of  the  design  configuration. 

8.  NAVSEA  (Human  Engineering  and  Manpower  and  Training  Analyst).  The  question 
of  who  will  apply  this  guide  is  paramount.  Given  a  program  office  which  is  organized  and 
functioning  to  produce  a  system,  the  engineers,  managers,  and  administrators  are 
committed  to  the  hardware  design  and  production.  Application  of  the  guide  will  require 
additional  workload  and,  hence,  labor  and  funding.  Application  of  the  objective  of  the 
guide  may  make  an  important  contribution  to  the  system;  however,  given  the  engineering 
orientation  and  technical  specialty,  someone  with  a  background  in  personnel,  manning, 
training,  and  engineering  will  need  to  be  added  to  the  program  office  to  implement  the 
objectives  of  the  guide. 

The  guide,  in  itself,  does  not  lend  itself  to  application  by  an  engineer.  The  guide 
falls  short  of  a  cookbook;  hence,  the  application  must  be  performed  by  someone 
competent  in  the  manpower  field  to  implement  its  objectives.  It  could  almost  be  said  that 
were  such  a  person  within  the  project  office,  the  guide  would  not  be  necessary.  It  is 
concluded  that  an  engineer  cannot  use  this  guide  to  more  effectively  design  a  system  for 
optimum  use  of  human  resources,  nor  is  the  guide  sufficient  to  determine  personnel  and 
training  requirements  more  effectively  than  is  currently  being  done.  The  value  of  the 
information  in  this  document,  if  recast,  may  be  to  convey  a  sharp  awareness  to  program 
managers  of  the  importance,  complexity,  and  costs  of  system  manpower.  The  type  of 
data  in  Sections  1  and  2,  dealing  with  the  impact  of  design  concepts  on  manning,  offers  a 
potential  to  have  a  very  worthy  impact  on  design.  Were  these  data  carefully  validated 
and  well-refined,  a  set  of  principles  and  rules  for  an  engineer  to  follow  for  effective  use 
of  personnel  could  be  formulated.  While  the  document  currently  falls  short  of  providing 
guidance  to  engineers  concerning  use  of  human  resources  in  system  design,  identification 
of  the  factors  involved  and  the  clustering  of  some  data  into  a  single  document  would  be  of 
use  to  manning  specialists  in  assisting  engineers  in  system  design. 

Implementing  the  Guide 

1.  NOSC  (Program  Manager).  Industry  is  the  place  that  the  guide  would  be  used. 
The  guide  has  to  become  a  requirement  in  the  contract.  If  OPNAV  implements  the  guide, 
NAVVIAT  will  have  to  follow  suit.  Just  making  it  a  NAVELEX  Standard  won't  give  it  the 
coverage  that  it  would  get  if  it  were  an  instruction  from  OPNAV  or  NAVMAT. 

2.  Contractor  (Senior  Staff  Engineer).  It  is  encouraging  to  see  serious  consideration 
given  to  this  important  area,  and  I  am  appreciative  of  the  opportunity  to  contribute  to 
this  effort.  I  hope  that  any  comments  and  evaluation  are  viewed  as  constructive  criticism 
of  the  content  or  procedures  and  not  seen  as  opposition  to  the  intent  of  the  guide. 


Although  the  guide  is  interesting  and  inforn  ,ve,  its  present  format  would  not 
be  expected  to  produce  any  appreciable  change  in  the  human  resources  requirements  of  a 
system  design  developed  and  procured  under  today's  environment  and  practices.  Some 
specific  suggestions  are:  After  thorough  quantitative  validation  of  the  evaluation  criteria 
profiles  (in  Section  2)  by  appropriate  organizations,  incorporate  the  applicable  ones  in  a 
new  military  handbook  on  human  resource  predictions.  Also,  validate  and  include  Sections 
6  and  7  in  the  new  military  handbook. 

Develop  minimum  human  resource  requirements  for  inclusion  in  contract 
specifications  and  for  award  evaluation  criteria.  General  requirements  would  have  to  be 
quantified  specifically  for  each  system  under  procurement.  By  this  procedure,  human 
resources  are  included  in  design  criteria,  first  as  incentivised  by  contract  award,  and 
second  by  quantitative  contract  requirements. 

3.  NAVSEA  (Branch  Head,  Manpower  and  Training  Analysis).  The  guide  should  be 
imposed  on  the  Program  Manager  and  Acquisition  Manager,  in  both  government  and  in  the 
contractor  structure.  This  guide  should  be  Source  Selection  Board  GF1.  Also,  it  should  be 
a  base  document  for  determining  contractor  capability  during  preaward  surveys.  Navy 
briefing  teams  should  provide  contractors  with  a  good  understanding  of  the  guide  and  its 
application. 

The  guide  should  be  a  military  handbook  that  must  be  referenced;  there  must  be 
an  "audit  trail"  to  ensure  that  the  guide  was  used  by  whomever. 

4.  Contractor  (Senior  Systems  Engineer).  The  unanimous  feeling  was  that  effective 
implementation  of  the  book  will  be  very  difficult  due  to  the  problem  of  establishing 
precise,  unambiguous,  measurable  tests  of  these  factors.  Once  such  tests  are  developed, 
they  should  be  funded  separately  to  ensure  the  task  receives  its  due. 

5.  NAVELEX  (Branch  Head).  The  guide  could  be  tied  in  to  logistics  support 
analysis:  Supply  support  and  maintenance  manning.  The  guide  is  more  applicable  to  the 
contractor  than  to  offices  in  NAVELEX.  The  acquisition  engineer  should  pull  out  parts  of 
the  guide  relevant  to  a  specific  system  and  include  it  in  the  contract  specification.  If  the 
guide  is  published  as  a  military  handbook,  it  can  be  invoked  as  either  a  guide  or  a 
requirement  in  the  contract  specification.  The  guide  should  carry  a  military  handbook 
number. 

6.  Contractor  (Project  Leader).  Ail  of  the  complicated  weighting  factors  could  be 
put  forth  by  the  customer  for  the  engineering  manager  at  the  contractor's  facility;  but  for 
a  designer,  the  data  were  not  readily  accessible.  There  is  no  index  to  guide  me  to  find 
particular  data.  It  was  too  much  like  an  academic  report. 

I  would  like  to  know  what  the  tradeoff  weights  are  between  manpower, 
performance,  size,  weight,  etc.  The  latter  three  are  what  drives  engineering  design; 
that's  what  we  are  trained  to  look  at.  If  you  want  to  establish  new  criteria  (manpower 
criteria),  you  must  provide  weights  or  some  basis  for  establishing  the  priorities  of  the  new 
criteria  over  the  old  ones  that  we  are  used  to. 

I  would  be  more  receptive  to  rules  of  thumb  and  straight  guidelines  than  pseudo¬ 
engineering  tradeoff  formulas. 

7.  NOSC  (Division  Head).  The  guide  will  only  work  if  the  guy  who  is  writing  the 
specifications  puts  it  in  the  requirements.  Unfortunately,  the  average  guy  writing  the 
specifications  can't  respond  to  the  guide.  Also,  no  one  pays  any  attention  to  the 


requirements  and  instructions;  the  Navy  doesn't  check  up  to  see  if  they  are  followed.  We 
need  a  method  that  is  an  economic  incentive  to  industry  to  follow  the  guide.  However, 
cost-incentive  programs  are  not  that  method,  they  just  don't  work. 

In  order  to  implement  the  guide,  you  must  train  acquisition  managers  in  the  use 
of  the  guide  so  that  they  will  make  it  a  requirement  to  contractors.  Subsequently,  you 
must  measure  the  contractor's  implementation  of  the  requirement.  Thus,  you  need  a 
deliverable  within  the  contract  that  is  related  to  the  guide. 

8.  Contractor  (Staff  Engineer).  The  underlying  USN  problem  is  reai  and 
appreciated;  however,  I  do  not  believe  your  draft  guide  will  solve  it.  If  the  Navy  intends 
to  make  consideration  of  human  resources  a  requirement  in  the  design  process,  then  it 
must  mandate  this  and  support  it  with  appropriate  resources. 

Suggestions  About  Guide  Content 

1.  Contractor  (Systems  Engineer,  Adequacy).  The  material  in  the  sections  follow¬ 
ing  Chapter  II  can  make  a  positive  contribution  to  electronics  system  design.  The  first 
two  chapters,  however,  should  be  reissued  to  reflect  more  emphatically  and  more 
accurately  what  the  proposed  technique  purports  to  accomplish.  I  found  the  introductory 
portions  weak. 

The  example  of  the  system  design  analysis  is  good,  but  it  seems  to  come  to  a 
screeching  halt  with  the  statement  that  the  consequent  design  improvement  would  then 
come  about  by  iteration.  This  a  key  part  of  the  guide  and  ought  to  be  elaborated  and 
emphasized  by  example.  What  follows  is  anticlimactic  and  fits  more  appropriately  into 
appendix  section. 

2.  Contractor  (Staff  Engineer).  If  designers  are  to  meet  a  new  mandatory 
requirement,  then  they  must  be  provided  with  a  "handbook"  which  is  simpler,  clearer,  and 
less  pedantic  than  the  guide.  The  guide's  logic  is  too  complicated  (e.g.,  Figure  3)  and  no 
beginning- to-end  example  is  presented  for  guidance  purposes.  The  guide  strikes  me  as  too 
academic  and  "researchy"  to  be  of  real  use  to  designers.  For  application  in  the  real  world, 
data  must  be  available  in  a  more  concise,  straightforward  form.  At  this  point,  it  is  more 
suited  for  program  managers. 

3.  Contractor  (Project  Leader).  The  academic  stuff  was  okay  for  an  introduction, 
but  it  did  not  facilitate  use  of  the  data  when  it  permeated  through  the  whole  book.  As  a 
designer  I  would  be  looking  more  for  a  handbook  rather  than  for  a  justification  as  to  why 
the  handbook  was  necessary. 

4.  Contractor  (Senior  Systems  Engineer).  Write  with  more  commonly  used  English, 
even  if  more  words  are  required.  The  biggest  example  seems  to  be  "taxonomy"  although 
"evaluative  criteria"  received  some  votes.  You  cannot  write  above  a  tenth  grade  level  if 
you  expect  systems  engineers  to  read  it. 

5.  NAVSEA  (Human  Engineering,  Manpower,  and  Training  Analyst).  It  is  recom¬ 
mended  that  the  guide  be  retailored  to  identify  those  features  which  contribute  to  the 
single  best  design  (i.e.,  a  set  of  rules  to  follow  to  optimize  the  design).  For  example,  from 
the  profiles  of  Section  1,  only  good  can  come  from  a  design  that  includes  equipment 
layout  to  facilitate  maintenance,  uses  standard  hardware  components,  and  "designs  for 
operational  simplicity";  but  nothing  good  can  be  said  about  "combined  opera tor/maintainer 
functions."  Don't  tell  the  engineer  to  "compare"  and  "tradeoff";  instead,  tell  him  how  to 
build  the  best  system  from  a  manning  and  training  standpoint  and  how  he  should  go  about 
it. 
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6.  Contractor  (Readiness  Design  Manager).  I  would  rather  have  seen  a  handbook 
called  "Guide  to  Concept  Selection  in  Electronic  Systems  for  Shipboard  Use,"  which 
contained  individual  sections  based  on  design  concepts.  This  would  put  forth  a  detailed 
description  of  a  particular  design  concept  followed  by  the  related  data.  For  example, 
program  development  considerations,  design  requirements,  manning  impact,  training 
impact,  fleet/ship  operations  impact,  and  examples/experience  reference  data. 

This  approach  would  have  two  major  advantages:  It  would  provide  a  guide  for 
the  selection  of  the  basic  concept,  applicable  to  both  Navy  and  contractor  use.  It  would 
be  easily  expandable  to  include  the  introduction  of  new  concepts  in  new  sections. 
Additional  data  could  easily  be  added  under  each  concept  section  in  terms  of  options, 
improvements,  and  experience  gained  in  the  future. 

7.  Contractor  (Program  Manager  and  staff).  The  guide  did  not  describe  the  problem 
and  the  exact  purpose  of  the  guide  very  clearly.  Suggest  you  list  problems  of  prior 
systems.  Discuss  generation  to  generation  problems;  use  tech-eval  and  op-eval  data. 
Don't  quantify  it,  just  highlight  the  important  points. 

Update  the  guide  yearly;  for  example,  the  Navy  Safety  Center  publishes  the 
Navy  Reactor  Reports  for  Nuclear  Operations.  These  reports  describe  very  succinctly  the 
problems  that  happen  from  time  to  time.  They  describe  personnel  actions  or  design  flaws 
that  may  have  caused  problems. 

The  guide  should  be  in  the  form  of  a  notebook  which  is  updated  yearly,  as 
discussed  above.  These  updates  would  also  tell  what  design  concepts  were  tried,  on  which 
systems,  and  how  the  design  concepts  worked  out.  This  would  provide  a  corporate 
memory  for  the  acquisition  and  design  community. 

There  is  a  35-year  gap  between  the  baseline  systems  that  were  used  for  the 
design  concepts  and  the  systems  that  are  being  designed  today  for  the  1985-1995  time 
frame.  Because  of  this  gap,  the  data  in  the  guide  don't  relate  at  all.  The  guide  needs 
current  projections  on  technical  trends,  manning  trends,  and  Navy  planning.  Use  systems 
that  are  currently  under  design,  being  tested  just  prior  to  fleet  introduction,  or  were 
recently  introduced.  Don't  use  systems  that  have  been  operationally  deployed  for  a  long 
period  of  time,  because  they  were  designed  in  the  '50s  and  '60s. 

Hit  logistics  and  training.  Answer  the  question,  "What  happens  when  the  system 
hits  the  fleet  during  the  introduction  period?" 

Checklists  and  rules  of  thumb  are  very  useful.  Would  prefer  rules  of  thumb  to  be 
presented  in  the  guide  instead  of  tradeoff  formulas.  Prefer  ballpark  estimates  on  costs 
(e.g.,  cost  billet  information  should  not  be  down  to  exact  dollars).  It  should  provide 
information  like,  "A  sailor  costs  $200,000  for  20  years."  Then  the  designer  and  acquisition 
people  can  tradeoff  number  of  cards,  modules,  etc.,  to  answer  the  question,  "Does  saving 
one  man  really  save  that  much  money?"  ; 

Suggestions  on  Guide  Format 

1.  NAVSEA  (Branch  Head  and  staff).  The  guide  is  too  cumbersome.  The  good 
material  in  the  guide  is  too  spread  out.  It  should  be  streamlined  for  engineers.  It  could  be 
broken  into  two  volumes:  The  first  volume  should  give  the  "quick  and  dirty"  answers  to 
the  problem.  The  second  volume  should  provide  backup  information  to  instill  confidence 
in  the  guide.  It  might  be  best  to  publish  Volume  I  and  Volume  II  in  a  single  document,  so 
that  they  do  not  get  separated  from  one  another;  however,  the  Volume  I  material  should 
be  on  colored  paper  in  the  front,  Volume  II  on  white  paper. 


Organize  the  whole  thing  by  functions  or  systems,  not  by  ratings.  Preface  each 
section  with  manpower  availability  charts  for  that  particular  type  of  equipment.  Within 
each  section,  devote  two  to  three  pages  per  example  system,  no  more. 

A  sample  application  of  the  guide  could  be  provided  in  an  appendix  to  Volume  I. 

The  guide  must  be  more  directive;  currently  it  does  not  tell  the  engineer  exactly 
what  to  do  with  the  information. 

Get  rid  of  the  word  "taxonomy."  Keep  terminology  in  designers'  language,  not  in 
terms  of  manning  of  ILS. 

Tab  or  index  the  material.  Use  a  three-ring  binder  or  screwpost  type  binder  in 
order  to  allow  updating  for  new  material  as  it  becomes  available.  At  the  very  least,  drill 
the  paper  for  a  three-ring  binder  and  send  it  out  stapled,  so  that  users  can  put  it  in 
whatever  binder  format  they  want. 

2.  Contractor  (Senior  Systems  Engineer).  Divide  the  book  into  two  parts:  One 
being  a  small  part  with  the  conceptual  information,  and  the  second  part  being  the  data 
section.  The  object  would  be  to  increase  the  likelihood  of  the  book  being  read. 

3.  NAVELEX  (Branch  Head  and  staff).  This  material  needs  a  roadmap  to  help 
people  get  through  the  information  in  the  guide.  Do  not  like  page  after  page  of  words; 
want  good  stuff  to  jump  out.  Improve  the  format  and  layout.  It  needs  to  be  human- 
factored! 


4.  NAVSEA  (Branch  Head,  Manpower  and  Training  Analysis).  The  guide  is  excel¬ 
lent,  an  innovative  format.  The  graphics  could  be  an  important  and  refreshing  input  to 
the  PM-AM  fiscal-statistical  analysis  supporting  program  presentation  before  DoD,  OMB, 
Senate  and  House  Conferences.  But  an  addition  is  needed.  A  radical,  simple,  usable, 
attactive,  eye-catching  device.  Perhaps  this  could  be  a  large  slide  rule  format,  which 
would  be  graduated  to  scale  the  events  along  the  critical  path.  Include:  stage  of 
development,  projected  personnel  supply,  tasks  and  skill  levels,  difficult  and  time 
consuming  tasks,  training  requirements  and  NECs,  and  billet  life  cycle  costs. 


CONCLUSIONS 


General 


1.  Most  respondents  from  the  Navy  system  acquisition  and  development  community 
believed  that  the  information  in  the  Engineer's  guide  would  benefit  the  system  design  and 
acquisition  process  by  making  human  resources  a  specific  design  consideration  and  by 
providing  necessary  technical  data. 

2.  In  general,  senior,  high-level  engineering  managers  from  the  contractor  design 
community  endorsed  applying  the  information  in  the  guide  to  the  system  design  and 
acquisition  process;  however,  mid-level  designers  often  had  difficulty  relating  the  guide  to 
their  immediate  design  problems. 

3.  A  subset  of  the  contractor  design  community  rejected  the  idea  of  designing  in 
response  to  manpower  requirements;  these  designers  believed  that  the  Navy  personnel  and 
training  system  should  solve  manpower  problems,  and  be  responsive  to  design,  not  the 
reverse.  Another  subset  of  designers,  however,  endorsed  the  concept  of  addressing 
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manpower  problems  at  the  system  definition  and  design  stage.  The  second  group  endorsed 
the  guide  and/or  wanted  to  see  it  improved. 


k.  The  current  form  of  the  guide  is  inappropriate;  it  does  not  allow  specification  of 
the  guide,  in  whole  or  part,  as  a  requirement  in  contracts.  A  military  instruction, 
handbook,  or  standard  would  be  a  much  more  appropriate  form. 

5.  The  information  in  the  guide  is  too  fragmented.  The  current  format  of  the  guide 
does  not  allow  efficient  and  appealing  access  to  important  data.  To  apply  information  in 
the  guide  to  the  design  of  a  particular  system,  a  user  would  have  to  gather  bits  and  pieces 
of  information  from  a  variety  of  places  in  the  guide.  The  guide  should  be  reorganized  by 
type  of  system. 

6.  The  guide  confounds  the  two  major  types  of  users.  Navy  program  managers  and 
system  acquisition  personnel  will  use  it  differently  than  contractor  managers  and 
designers.  The  guide  must  be  explicit  on  how  each  group  will  apply  the  information.  The 
organization  of  material  in  the  guide  must  facilitate  the  two  different  types  of  use. 

7.  Titles,  headings,  and  introductory  comments  should  not  purport  to  cover  a  larger 
scope  than  the  guide  actually  does.  For  example,  many  of  the  sections  in  the  guide  do  not 
cover  the  topics  as  completely  as  implied  by  the  introductions.  Also,  the  title  of  the 
guide  should  delimit  the  scope  of  the  document,  that  it  is  restricted  to  five  types  of 
systems  aboard  surface  ships. 

8.  The  varied  backgrounds  of  the  guide's  users  require  that  a  glossary  be  added  to 
ensure  universal  understanding  of  technical  terms  in  the  guide. 

9.  A  corollary  to  the  conclusion  above  is  that,  as  much  as  possible,  technical  terms 
and  jargon  be  reduced  to  their  common- language  equivalents.  The  biggest  offender  was 
"taxonomy,"  although  the  detailed  comments  suggested  other  problem  terms. 

10.  An  index  is  needed  for  efficient  access  and  retrieval  of  information.  Re¬ 
organization  of  the  guide  will  reduce  but  not  eliminate  the  need  for  an  index. 

Specific  Sections 

l.  The  two  introductory  chapters  are  inadequate;  they  do  not  introduce  the 
remainder  of  the  guide  very  well  nor  do  they  provide  accurate  direction  on  specific 
requirements,  and  goals  to  be  achieved  as  a  result  of  using  the  guide. 

2.  The  second  introductory  chapter  is  naive.  The  premise  underlying  system  design, 
exemplified  in  Figure  3,  is  an  ideal  that  does  not  relate  well  to  the  real  design  world:  The 
notion  of  iterative  solutions  to  design,  with  feedback  loops  from  later  points  in  the  overall 
design  process  to  earlier  ones  is  not  realistic.  Except  for  certain  large  system 
acquisitions,  integrated  ship  designs,  and,  perhaps,  some  laboratory  development  efforts, 
conceptual  design  is  a  one-step  process  where  contractor  designers  take  their  "best  shot" 
at  design  proposal  and  preproposal  stages.  To  introduce  the  guide  in  the  context  of  a 
grand,  tradeoff-based,  iterative  process  will  jeopardize  the  guide's  credibility  with  many 
experienced  designers. 

3.  The  data  on  design  concepts  are  potentially  useful  for  designing  systems  to 
require  fewer  human  resources.  However,  the  current  presentation  and  data  base  have 
problems  that  need  to  be  solved.  Specific  conclusions  regarding  the  design  concept  data 
base  in  Section  1  include: 
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a.  Design  concepts  must  take  into  account  types  of  equipment  and  types  of 
missions.  This  is  very  critical  for  considering  built-in  test  equipment,  and  the  group  of 
design  concepts  related  to  sparing  strategy. 

b.  Definitions  of  design  concepts  need  to  be  improved;  they  should  be  il¬ 
lustrated  with  more  detail  and  examples. 

c.  The  judgmental  data  base  needs  to  be  improved.  Data  need  to  be  gathered 
from  a  larger  group  of  judges.  Judges  must  have  specific  credentials  qualifying  them  to 
relate  the  impact  of  design  concepts  on  specific  system  design  criteria.  The  numerous 
indecision  marks  in  the  current  data  base  must  be  resolved;  they  seriously  jeopardize  the 
credibility  of  the  data. 

d.  The  measurement  scale  relating  impact  of  design  concepts  on  system  design 
criteria  must  be  improved  to  make  this  information  applicable  during  design.  A 
translation  of  this  scale  into  terms  of  cost  would  be  one  welcomed  solution. 

e.  The  set  of  design  concepts  concerning  LRUs,  spares,  and  location  of  repair 
should  be  reorganized  and  consolidated  to  better  reflect  basic  tradeoffs  in  this  area. 
LRUs-No  spares  should  probably  be  dropped  entirely. 

f.  The  treatment  of  embedded  computers  must  be  entirely  revised  to  better 
reflect  the  true  nature  of  this  technical  area  and  the  current  trends  in  large  scale 
integration  (LSI)  and  very  large  scale  integration  (VLSI). 

g.  The  design  concepts  concerning  various  levels  of  standard  hardware  need  to 
be  examined  and  possibly  reorganized  and  consolidated  to  better  reflect  viable 
alternatives  in  this  area. 

h.  All  of  the  definitions  of  design  concepts  should  be  examined,  rewritten,  and 
edited  with  the  assistance  of  engineering  design  experts. 

i.  The  additional  design  concepts  suggested  by  commentators  need  to  be 
examined  and  screened  for  inclusion  in  a  revised  guide,  again  with  the  assistance  of 
engineering  design  experts. 

4.  The  reorganization  of  Section  1  information  and  the  presentation  of  it  in 
Section  2  was  confusing  to  some  evaluators.  At  the  least,  definitions  of  system  design 
criteria  should  be  presented  along  with  the  definitions  of  design  concepts;  the  present 
separation  of  these  definitions  makes  Section  1  and  Section  2  incomplete  and  hard  to 
understand  by  themselves.  Further,  consolidation  of  the  data  represented  in  the  profiles 
may  be  very  helpful  (e.g.,  eliminating  variables  on  a  profile  that  have  a  negligible  impact, 
presenting  the  data  in  discrete  categories  in  tabular  form).  Other  specific  conclusions 
regarding  system  design  criteria  in  Section  2  include: 

a.  Each  system  design  criterion  should  be  examined,  rewritten,  and  edited  with 
the  assistance  of  engineering  design  experts. 

b.  The  guide  must  carefully  distinguish  between  MTTR  (Mean  Time  To  Repair) 
and  MLDT  (Mean  Logistics  Down  Time)  and,  possibly,  use  the  latter  as  a  more  appropriate 
criterion  for  trading  off  the  overall  effects  of  the  design  concepts. 

c.  Suggested  additional  system  design  criteria  should  be  examined  and  screened 
for  possible  inclusion  in  the  guide,  with  the  assistance  of  engineering  design  experts. 
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d.  The  judges'  expertise  and  subjective  data  were  especially  suspect  regarding 
the  intricacies  of  supply  and  support  costs.  All  of  the  impacts  of  hardware  standardiza¬ 
tion  concepts  and  level  of  repair  concepts  on  supply  and  support  costs  were  questionable. 
This  aspect  of  the  data  base  especially  needs  improvement. 

5.  The  material  in  Section  3  on  types  of  technicians  assigned  to  surface  ship 
electronic  systems  was  completely  inadequate.  The  large  number  of  comments  that 
suggested  additional  types  of  information  to  be  included  in  this  section  indicate  that 
designers  need,  and  are  interested  in,  descriptive  information  on  Navy  technicians.  More 
complete  descriptions  of  each  type  of  technician  should  be  developed  for  the  revised 
guide. 


6.  If  the  information  in  Section  4  on  projected  supply  of  technical  ratings  is 
included  in  the  revised  guide,  the  specific  users  must  be  identified  and  provided  with 
instructions  for  applying  the  information  to  either  system  acquisition  or  design.  In  its 
current  form,  the  information  is  not  perceived  to  be  relevant  or  useful  to  designers. 

7.  A  small  but  articulate  group  of  commentators  have  argued  against  the  utility  of 
the  evaluation  of  alternatives  approach  in  Section  5.  The  approach  has  been  called 
"pseudo-engineering,"  an  attempt  to  derive  a  superficially  refined  comparison  index  based 
upon  relatively  crude  judgmental  data.  The  underlying  data  have  not  been  validated  and 
there  is  very  likely  a  wide  variability  in  the  applicability  of  the  data  to  different  types  of 
systems,  for  different  types  of  missions,  etc.  Further,  the  notion  of  comparing 
alternative  system  designs  has  been  called  unrealistic  (see  conclusions  regarding  intro¬ 
ductory  material  in  Chapter  II);  several  commentators  stated  that  a  designer  hardly  has 
sufficient  time  and  resources  to  adequately  design  one  system,  let  alone  several  for 
comparison.  A  sizable  number  of  comments  suggested  using  a  checklist  to  present 
straightforward  guidelines  for  system  design. 

8.  The  applicability  and  utility  of  data  in  Sections  6  and  7  for  improving  design  will 
be  dependent  upon  two  things.  One  is  a  detailed  specification  of  how  the  designer  is  to 
apply  information  about  old  systems  to  the  design  of  new  ones.  The  other  is  a  need  to 
base  the  historical  data  on  systems  under  development  or  very  recently  deployed  to  the 
fleet;  many  commentators  believed  that  the  systems  included  in  Sections  6  and  7  were  too 
old  to  be  relevant  to  future  design  efforts.  Other  conclusions  regarding  Sections  6  and  7 
include: 


a.  Brief  descriptions  of  example  systems  should  be  provided  as  a  technical 
context  for  the  task  data. 

b.  It  may  be  unnecessary  to  break  down  proficiency  profiles  by  first,  second, 
and  third  class  petty  officers,  because  the  crux  of  design  improvement  will  be  the 
elimination  or  improvement  of  problem  tasks,  which  will  improve  proficiency  of  all 
paygrades  of  technicians.  However,  commentators  who  expressed  this  opinion  also  wanted 
a  summary  of  the  general  differences  between  the  paygrades  and  a  statement  of  the 
probable  mix  of  paygrades  that  would  be  assigned  to  a  given  type  of  system.  This  would 
help  them  envision  the  level  of  maintenance  and  operation  typically  available  for  their 
system. 


9.  The  information  in  Section  8  on  training  requirements  and  NECs  is  inadequate. 
If  this  type  of  material  is  to  be  retained  in  the  guide,  additional  data  should  be  included  to 
make  it  useful. 
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10.  The  information  in  Section  9  on  billet  life  cycle  costs  is  useful  to  designers, 
although  additional  data  would  be  helpful.  The  level  of  precision  in  the  current  data 
tables  is  not  necessary. 

11.  Reference  material  should  be  expanded  and  annotated. 

RECOMMENDATIONS 

1.  The  specific  shortcomings  noted  in  the  current  guide,  which  are  indicated  in  the 
previous  section,  should  be  corrected. 

2.  The  guide  should  be  reorganized  and  reformatted.  Although  this  point  is 
subsumed  by  the  recommendation  to  correct  specific  shortcomings,  it  is  so  important  that 
it  should  be  made  a  major  recommendation.  The  recommended  preliminary  outline  will 
accommodate  the  majority  of  comments  regarding  organization  and  format. 

3.  Revision  of  the  guide  should  include  inputs  from  Navy  acquisition  organizations 
and  contractor  designers.  As  will  be  evident  in  the  recommended  preliminary  outline, 
many  topics  require  intimate  familiarity  with  current  acquisition  procedures  or  design 
technology.  Collaboration  with  experts  in  acquisition  and  design  will  be  necessary  to 
ensure  the  guide's  utility  and  eventual  acceptance  and  implementation  by  members  of  the 
user  communities.  In  addition  to  providing  guidance  and  technical  input  to  draft  revisions 
of  the  guide,  experts  from  the  acquisition  and  design  communities  should  help  edit  and 
check  portions  of  the  revised  guide  that  treat  their  areas  of  technical  expertise.  For 
example,  one  or  two  senior  contractor  designers  with  extensive  experience  in  radar 
systems  should  edit  the  section  on  radar.  One  or  two  experienced  directors  or  branch 
heads  from  appropriate  SYSCOMs  should  edit  material  that  specifies  how  the  guide  is  to 
be  used  by  the  acquisition  community. 

k.  The  revised  guide  must  be  a  military  instruction,  handbook,  or  standard.  This 
point  is  subsumed  by  the  recommendation  to  correct  shortcomings  in  the  guide;  however, 
it  is  so  important  that  it  must  be  made  a  major  recommendation.  In  its  current  form  as  a 
report,  the  guide  cannot  be  implemented  as  a  requirement  by  the  acquisition  community, 
and  it  would  be  unrealistic  to  think  that  the  design  community  would  apply  it  voluntarily. 
The  revised  guide  should  be  in  a  form  that  can  be  called  out  as  a  contractual  specification 
or,  at  the  very  least,  as  one  of  the  weighted  factors  in  proposal  evaluation  or  in  the  sole 
source  selection  process. 

5.  Alternative  mediums  for  the  guide  (e.g.,  three-ring  notebook,  computer-accessed 
data  base)  should  be  examined.  The  medium  for  the  revised  guide  must  allow  convenient 
and  cost-effective  updating,  because  much  of  the  most  useful  material  is  time-sensitive. 

A  preliminary  outline  for  the  revised  guide,  which  is  based  upon  consideration  and 
interpretation  of  all  of  the  data  that  were  gathered  during  the  evaluation,  is  presented  in 
Appendix  C.  This  outline  meets  the  letter  and  spirit  of  the  bulk  of  conscientious 
suggestions  that  were  received  during  this  evaluation. 
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APPENDIX  A 


EVALUATION  PACKAGE  C 


EVALUATION  PACKAGE  C  FOR: 

AN  ENGINEERS  GUIDE  TO  THE  USE  OF 
HUMAN  RESOURCES  IN 
ELECTRONIC  SYSTEMS  DESIGN 


Produced  for: 

NAVY  PERSONNEL  RESEARCH  AND  DEVELOPMENT  CENTER 
San  Diego,  California  92152 


Produced  by: 

ANACAPA  SCIENCES,  INC. 

Santa  Barbara,  California  93102 


February  1980 


A  MESSAGE  TO  EVALUATORS 


Greetings!  You  have  received  a  copy  of  this  evalua¬ 
tion  package  through  a  long  and  intricate  selection  process 
that  started  months  ago  in  San  Diego,  California.  Hopefully 
you  are  a  systems-level  designer  who  has  had  prior  experience 
in  the  design  and  development  of  electronic  systems  for  the 
U.S.  Navy.  If  you  have  worked  in  the  areas  of  radar,  sonar, 
fire  control,  communications,  or  data  processing,  that  is 
even  better. 

If  you  have  had  none  of  the  above  experience,  then 
someone  made  a  mistake  in  giving  this  package  to  you.  Please 
return  it  without  further  obligation. 

If  you  are  one  of  the  people  that  we  are  interested  in, 
please  read  on.  As  previously  arranged,  a  $100  honorarium 
will  be  given  to  qualified  evaluators  who  complete  this  package. 


THE  PROBLEM 

(IN  AN  OVERSIMPLIFIED  NUTSHELL' 

The  Navy  does  not  have  enough  men  to  operate  and  maintain  all 
of  its  electronic  systems;  a  shockingly  large  proportion  of  the  men 
that  are  available  don't  have  the  skill  and  experience  to  do  the  job 
(especially  maintenance)  correctly.  We  have  studied  this  problem  to 
death  and  most  of  the  conclusions  suggest  that  system  designers  could 
help  us  solve  it. 

Certain  cases  have  shown  that  systems  be  designed  to  be  op¬ 
erated  and  maintained  by  fewer  and  lower  skilled  men.  But  we  don't 
know  enough  about  how  to  coordinate  the  Navy  acquisition  process  with 
the  contractor  design  process  to  achieve  this  reliably  with  every  new 
system. 
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THE  ENGINEERS  GUIDE 

Development  of  the  "Engineers  Guide  to  the  Use  of  Human  Resources 
in  Electronic  Systems  Design"  was  an  initial  attempt  to  provide  the  kinds 
of  data  that  Navy  acquisition  management  teams  and  contractor  designers 
could  use  to  cope  with  manpower  problems.  Some  sections  of  the  report 
pinpoint  manpower  shortages  and  skill  deficiencies  for  specific  types  of 
electronic  systems.  Other  sections  present  data  and  procedures  for  eval¬ 
uating  manpower  and  cost  implications  of  alternative  system  designs  durinq 
early  stages  of  concept  development. 


UNCLASSIFIED 


ANACAPA  SCIENCES  INC  SANTA  BARBARA  CA  F/G  5/5 

ENGINEER'S  GUIDE  To  THE  USE  OF  HUMAN  RESOURCES  IN  ELECTRONIC  SY— ETC(U) 
NOV  80  R  A  DICK,  E  A  KOEHLER  N00123-79-C-1335 

NPRDC-SR-01-3  NL 


MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BUREAU  Of  STANDARDS  1963  A 


REVIEWING  THE  GUIDE 

Your  first  task  is  to  read  through  the  Guide  at  least  once  to 
grasp  the  overall  concept  and  to  learn  how  to  use  the  data  tables 
and  charts.  As  you  read  through  the  Guide,  please  ask  yourself  the 
following  questions: 

■  "Could  I  have  applied  this  information  to  the  last  Navy 
electronic  system  that  I  worked  on?" 

■  "Could  I  use  this  information  to  reduce  the  numbers  and 
skill  levels  of  operators  and  maintainers  of  future  systems." 
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EVALUATING  THE  GUIDE 

Your  second  task  is  to  go  through  the  Guide  section  by 
section  as  you  work  through  the  yellow  evaluation  forms.  The 
evaluation  forms  ask  for  two  kinds  of  information:  the  rating 
scales  and  checksheets  on  the  left-hand  pages  ask  for  struc¬ 
tured  responses  which  can  be  analyzed  statistically.  The 
right-hand  pages  ask  for  your  suggestions  for  modifying  the 
Guide  and  the  explanations  of  your  views. 

The  statistical  data  from  the  left-hand  pages  will  be 
used  to  pinpoint  problem  areas  in  the  Guide.  The  explanation 
data  from  the  right-hand  pages  will  be  used  for  making  modi¬ 
fications  to  the  problem  areas. 


USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTER  CAN  ANALYZE  STATISTICALLY 


Chapter  I:  Introduction  and  Chapter  II:  Designing  in  Relation  to  Human  Resources 

Circle  the  scale  number  that  best  indicates  your  degree  of  response  to  each  item.  Put 
an  "X"  in  the  "Don't  Know"  box  for  items  to  which  you  cannot  respond. 


C-l.  Considering  my  present  job  as  a  designer,  the  information  in  Chapters  I  &  II  is: 

I - ’ - , - 2 - |_J - , - 2 - t - 5— ,  [2 
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Useful 


Completely 

Useless 
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If  DOD  were  to  seriously  impose  design  constraints  based  on  manpower  considera¬ 
tions,  the  information  in  Chapters  I  &  II  would  be: 
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The  information  in  Chapters  I  &  II  appears  to  be: 
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The  information  in  Chapters  I  &  II  is: 
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&  Complete 
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The  format  (tables,  charts,  text,  etc.)  for  presenting  the  key  information 
in  Chapters  I  &  II  is: 
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In  order  to  provide  helpful  guidance  to  designers  like  me, 
should  be: 

i  1  i  2  ,  3 _ , _ 4 _ , _ 5 


Chapters  I  &  II 
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(or  Omitted) 
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USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTES  CAN  ANALYZE  STATISTICALLY 


Section  1:  Definition  and  Impact  of  Design  Concepts 

Circle  the  scale  number  that  best  indicates  your  degree  of  response  to  each  item.  Put 
an  "X"  in  the  "Don't  Know"  box  for  items  to  which  you  cannot  respond. 


1-1. 


Considering  my  present  job  as  a  designer,  the  information  in  Section  1  is: 
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If  DOD  were  to  seriously  impose  design  constraints  based  on  manpower  considera¬ 
tions,  the  information  in  Section  1  would  be: 
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1-3.  The  information  in  Section  1  appears  to  be: 
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1-4.  The  information  in  Section  1  is: 

,1,2,3 

| - j - = - ! - 

Thorough 
&  Complete 


Full  of  Gaps 
and  Omissions 


[  ] 

Don '  t 
Know 


1-5. 


The  format  (tables,  charts,  text,  etc.)  for  presenting  the  key  information 
in  Section  1  is: 


1 _ i _  2,3  ,  4  ,  5  .  C  ] 


Appropriate; 

Easy  to 
Understand 


Inappropriate; 
Hard  to 
Understand 


Don't 

Know 


1-6. 


In  order  to  provide  helpful  guidance  to  designers  like  me.  Section  1 
be: 


Kept  exactly 
as  is 


Changed 
Drastically 
(or  Omitted) 


should 

[  ] 

Don't 

Know 
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USE  THIS  PAGE  TO  PROVIDE  EXPLANATIONS  THAT  OUR  HUMAN 
ANALYST  CAN  USE  TO  MODIFY  AND  IMPROVE  THE  GUIDE 

Section  1:  Definition  and  Impact  of  Design  Concepts 

Explain  your  reasons  for  circling  a  "4"  or  ”5“  on  any  item  on  the  opposite  page. 


Use  additional  sheets  for  long  explanations.  Remember  to  indicate  the  Item  number. 

A-ll 


USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTER  CAN  ANALYZE  STATISTICALLY 


Design  Concepts:  Place  an  "X"  in  one  of  the  two  columns  under  A  and  an  "X"  in  one  of 
the  three  columns  under  B  to  express  your  opinion  of  each  Design 
Concept  discussed  in  Section  1  of  the  Guide. 

- A - 1  i - B - 


I  Don't 

I  Under-  Under  USEFUL:  lUSEFUL:  IUSELESS 


1.  Equipment  layout  to  facilitate  maintenance 


2.  LRUs— No  spares 


3.  LRUs— Spares  with  onboard  repair 


4.  LRUs— Spares  with  remote  repair 


5.  LRUs— Spares  with  throwaway  maintenance 


6.  "Overdesign"  for  reliability  &  maintenance 


7.  Embedded  computers 


8.  Automatic  performance  monitoring 


9.  Built-in  test  equipment 


10.  Built-in  troubleshooting  logic  aids 


11.  Automatic  fault  localization 


12.  Standard  hardware  components 


13.  Standard  hardware— Cards/LRUs 


14.  Standard  hardware— Functional  units 


15.  Standard  hardware-Subsystems 


16.  Operational  simplicity 


17.  Built-in  operator  performance  aids 


18.  Automatic  decision  making 


19.  Automatic  information  transmit  &  display 


20.  Built-in  training  capability 


21.  Combined  operator/malntainer  functions 


I  I  Additional  Design  Concepts:  Place  an  "X"  in  the  box  at  the  left  if  you  believe 

that  Design  Concepts  which  could  significantly  impact 
manpower  have  been  left  out  of  Section  1  of  the  Guide. 


USE  THIS  PAGE  TO  PROVIDE  EXPLANATIONS  THAT  OUR  HUMAN 
ANALYST  CAN  USE  TO  MODIFY  AND  IMPROVE  THE  GUIDE 


Design  Concepts: 


Design  Concept 
Number 


Explain  your  thoughts  on  the  5  Design  Concepts  on  the  opposite 
page  which  you  most  strongly  believe  should  be  modified,  or 
should  be  dropped.  Please  write  the  Design  Concept  number  be¬ 
fore  each  explanation. 


Explain  any  Design  Concepts  that  should  be  added  to  the  list. 


Use  additional  sheets  for  long  explanations.  Remember  to  indicate  the 
Design  Concept  number.  (Use  X-l,  X-2,  etc.,  for  new  ones.) 


USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTER  CAN  ANALYZE  STATISTICALLY 

Section  2.  Interaction  of  Design  Concept  Impacts  on  Different  System  Design  Criteria 

Circle  the  scale  number  that  best  indicates  your  degree  of  response  to  each  item.  Put 
an  "X"  in  the  "Don't  Know"  box  for  items  to  which  you  cannot  respond. 


2-1.  Considering  my  present  job  as  a  designer,  the  information  in  Section  2  is: 

— i — i — i — 3 — , — — 5 — i  r  i 

Don't 
Know 


Very 

Useful 


Completely 

Useless 


2-2.  if  DOD  were  to  seriously  impose  design  constraints  based  on  manpower  considera¬ 
tions,  the  information  in  Section  2  would  be: 


1  1  I 

2  1-  3  -4  4 

1  5  1 

[  3 

1  1 

Very 

Useful 

1  1 

1  1 
Completely 
Useless 

Don't 

Know 

2-3.  The  information  in  Section 

1  1  1 

2  appears  to  be: 
2,3,4 

|  5  , 

[  3 

1  t 

Very 

Accurate 

I  I 

1  1 

Very 

Questionable 

Don't 

Know 

2-4.  The  information  in  Section  2  is: 

1  .  2 


b 

Thorough 
&  Complete 


H - 2 - 1 

Ful 1  of  Gaps 
and  Omissions 


[  ] 

Don't 

Know 


2-5.  The  format  (tables,  charts,  text,  etc.)  for  presenting  the  key  information 
in  Section  2  is: 

[  ] 


be: 


\  1  !  2  j  3  i  4 

|  5  j 

Appropriate; 

Inappropriate; 

Easy  to 

Hard  to 

Understand 

Understand 

provide  helpful  guidance  to  designers 

like  me.  Section 

I - ! - _| - 1 - 1 - 2 - , - 4_ 

< 

Don't 

Know 


Kept  exactly 
as  is 


Changed 
Drastically 
(or  Omitted) 


C  ] 

Don't 

Know 


A-14 


USE  THIS  PAGE  TO  PROVIDE  EXPLANATIONS  THAT  OUR  HUMAN 
ANALYST  CAN  USE  TO  MODIFY  AND  IMPROVE  THE  GUIDE 


Section  2:  Interaction  of  Design  Concept  Impacts  on  Different  System  Design  Criteria 
Explain  your  reasons  for  circling  a  "4"  or  "5"  on  any  item  on  the  opposite  page. 


2-1. 


2-2. 


2-3. 


2-4. 


2-5. 


2-6. 


Use  additional  sheets  for  long  explanations.  Remember  to  indicate  the  item  number. 

A-15 


USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTER  CAN  ANALYZE  STATISTICALLY 


System  Design  Criteria:  Place  an  "X“  in  one  of  the  two  columns  under  A  and  an  "X"  in 

one  of  the  three  columns  under  B  to  express  your  opinion  of  each 
System  Design  Criterion  discussed  in  Section  2  of  the  Guide. 


I  Under¬ 
stand 
it 

I  Don't 
Under¬ 
stand 
it 

USEFUL: 
Keep  it 
as  is 

USEFUL: 

Modify 

it 

USELESS: 

Drop 

it 

A. 

Maintainer  skill  and  experience  level 
required 

B. 

System- specific  maintenance  training 
requ i red 

C. 

Shipboard  maintenance  man-hours  required 

D. 

MTBF 

E. 

MTTR 

F. 

Tools,  test  equipment,  facilities  costs 

G. 

Supply  &  support  costs 

H. 

Operator  skill  &  experience  level  required 

I. 

System-specific  operator  training  required 

J. 

Total  number  of  operators  required 

K. 

System  operability 

L. 

Overall  operational  capability  & 
effectiveness 

M. 

Initial  system  acquisition  costs 

N. 

Operational  lifetime  costs 

1 

I  I  Additional  System  Design  Criteria:  Place  an  "X"  in  the  box  at  the  left  if  you  believe 

that  important  System  Design  Criteria  have  been 
left  out  of  Section  2  of  the  Guide. 


A-1  6 


USE  THIS  PAGE  TO  PROVIDE  EXPLANATIONS  THAT  OUR  HUMAN 
ANALYST  CAN  USE  TO  MODIFY  AND  IMPROVE  THE  GUIDE 

System  Design  Criteria:  Explain  your  thoughts  on  the  5  System  Design  Criteria  on  the 

opposite  page  which  you  most  strongly  believe  should  be 
modified,  or  should  be  dropped.  Please  write  the  System 
Design  Criterion  letter  before  each  explanation. 


System  Design 
Criterion  Letter 


Explain  any  System  Design  Criteria  that  should  be  added  to  the  list. 


Use  additional  sheets  for  long  explanations.  Remember  to  indicate  the 
System  Design  Criterion  letter.  (Use  X-A,  X-B,  etc.,  for  new  ones.) 


A-17 


USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTER  CAN  ANALYZE  STATISTICALLY 


Section  3:  Types  of  Technicians  Assigned  to  Surface  Ship  Electronic  Systems 

Circle  the  scale  number  that  best  indicates  your  degree  of  response  to  each  item.  Put 
an  "X"  in  the  "Don't  Know"  box  for  items  to  which  you  cannot  respond. 


3-1. 

Considering  my  present  job  as  a  designer,  the 

information  in  Section 

3  is: 

1  1  1  2  1-  3-  1 

4 

|  .  5  , 

[  ] 

1  1  1  1 
Very 

s  1 

Completely 

Don't 

Useful 

Useless 

Know 

3-2. 

If  DOD  were  to  seriously  impose  design  constraints  based  on  manpower 

consic 

tions,  the  information  in  Section  3  would  be: 

,  1,2,3  , 

4 

__i _ § _ | 

[  1 

ili! 

Very 

1  1 
Completely 

Don't 

Useful 

Useless 

Know 

3-3. 

The  information  in  Section  3  appears  to  be: 

,  1,2  ,3.  , 

4 

L _ 5 _ J 

C  ] 

1  1  t  1 

Very 

1  ! 

Very 

Don't 

Accurate 

Questionable 

Know 

3-4. 

The  information  in  Section  3  is: 

4 

1  -  5  1 

[  ] 

till 

Thorough 

1  l 

Ful 1  of  Gaps 

Don't 

&  Complete 

and  Omissions 

Know 

3-5. 

The  format  (tables,  charts,  text,  etc.)  for  presenting  the  key  information 

ir.  Section  3  is: 

I  1  1-  2  L  3  I 

4 

1  5  1 

[  ] 

,1.1  >  1 
Appropriate; 

I  1 

Inappropriate; 

Don't 

Easy  to 

Hard  to 

Know 

Understand 


Understand 


3-6. 


In  order  to  provide  helpful  guidance  to  designers  like  me,  Section  3  should 
be : 

1  ,2  ,3  ,4,5 


h 


-f 


Kept  exactly 
as  is 


Changed 
Drastically 
(or  Omitted) 


[  ] 

Don't 

Know 


2 


3 


4 


USE  THIS  PAGE  TO  PROVIDE  EXPLANATIONS  THAT  OUR  HUMAN 
ANALYST  CAN  USE  TO  MODIFY  AND  IMPROVE  THE  GUIDE 

Section  3.  Types  of  Technicians  Assigned  to  Surface  Ship  Electronic  Systems 

Explain  your  reasons  for  circling  a  "4"  or  "5"  on  any  item  on  the  opposite  page. 

3-1.  _  _ 


Use  additional  sheets  for  long  explanations.  Remeirtber  to  indicate  the  item  number 


USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTER  CAN  ANALYZE  STATISTICALLY 


Section  4:  Projected  Supply  of  Technical  Ratings  at  Different  Experience  Levels 

Circle  the  scale  number  that  best  indicates  your  degree  of  response  to  each  item.  Put 
an  "X"  in  the  "Don't  Know"  box  for  items  to  which  you  cannot  respond. 


4-1.  Considering  my  present  job  as  a  designer,  the  information  in  Section  4  is: 

i  1  i  -2  . i  -3  -  i-  *■  -i  5  i  n 

Very  Completely  Don  t 

Useful  Useless  Know 


Completely 

Useless 


If  DOD  were  to  seriously  impose  design  constraints  based  on  manpower  considera¬ 
tions,  the  information  in  Section  4  would  be: 


Very 

Useful 


Completely 

Useless 


Don't 

Know 


The  information  in  Section  4  appears  to  be: 

i  '■  >--2  —  M — h 

Very 

Accurate 


Very 

Questionable 


Don't 

Know 


The  information  in  Section  4  is: 


Thorough 
&  Complete 


Full  of  Gaps  Don't 

and  Omissions  Know 


The  format  (tables,  charts,  text,  etc.)  for  presenting  the  key  information 
in  Section  4  is: 

i  ■  .  i  2  i  3 — i — - — i — - — i  ” 

Appropriate;  Inappropriate;  Don't 

Easy  to  Hard  to  Know 

Understand  Understand 


In  order  to  provide  helpful  guidance  to  designers  like  me.  Section  4  should 
be: 


Kept  exactly 
as  is 


Changed 
Drastically 
(or  Omitted) 


Don't 

Know 


A-20 


USE  THIS  PAGE  TO  PROVIDE  EXPLANATIONS  THAT  OUR  HUMAN 
ANALYST  CAN  USE  TO  MODIFY  AND  IMPROVE  THE  GUIDE 


Section  4.  Projected  Supply  of  Technical  Ratings  at  Different  Experience  Levels 
Explain  your  reasons  for  circling  a  "4"  or  "5"  on  any  item  on  the  opposite  page. 


A-21 


USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTER  CAN  ANALYZE  STATISTICALLY 

Section  5.  Evaluation  of  Alternatives 

Circle  the  scale  number  that  best  indicates  your  degree  of  response  to  each  item.  Put 
an  "X"  in  the  "Don't  Know"  box  for  items  to  which  you  cannot  respond. 


5-1. 

Considering  my  present  job  as  a  designer,  the  inf 
.  1  ,2,3  ,4 

ormaticn  in  Section 

1  5  1 

5  is: 

C  ] 

1  1  1  1 

Very 

I  1 

Completely 

Don't 

Useful 

Useless 

Know 

5-2. 

If  DOD  were  to  seriously  impose  design  constraints  based  on  manpower 

considera 

ticns,  the  information  in  Section  5  would  be: 

11,2,3  1  4 

[  ] 

(ill 

Very 

1  1 
Completely 

Don't 

Useful 

Useless 

Know 

5-3. 

The  information  in  Section  5  appears  to  be: 

1  1  |  2  ,  3  ,4 

_  |  5  , 

[  ] 

1  !  !  1 

Very 

1  1 

Very 

Don't 

Accurate 

Questionable 

Know 

5-4. 

The  information  in  Section  5  is: 

j _ ] _ | _ 1 _ |  3  ■  4 

[  ] 

1  !  J  I 

Thorough 

- 1 

Full  of  Gaps 

Don't 

&  Complete 

and  Omissions 

Know 

5-5. 

The  format  (tables,  charts,  text,  etc.)  for  presenting  the  key  information 

in  Section  5  is: 

1 _ 1 _ [ —  2  |  3  1  4 

1  5  1 

[  ] 

J  1  l  1 

Appropriate; 

Inappropriate; 

Don't 

Easy  to 

Hard  to 

Know 

5-6. 

Understand 

In  order  to  provide  helpful  guidance  to  designers 

Understand 

like  me.  Section  5 

should 

be: 

i.  1  _i_  2,3,4 

1  5  1 

[  ] 

1  T  f  | 

Kept  exactly 

t  1 

Changed 

Don't 

as  is 

Drastically 

Know 

(or  Omitted) 
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USE  THIS  PAGE  TO  PROVIDE  EXPLANATIONS  THAT  OUR  HUMAN 
ANALYST  CAN  USE  TO  MODIFY  AND  IMPROVE  THE  GUIDE 

Section  5.  Evaluation  of  Alternatives 

Explain  your  reasons  for  circling  a  "4"  or  ''5"  on  any  item  on  the  opposite  page. 


Use  additional  sheets  for  long  explanations.  Remember  to  indicate  the  item  number. 


USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTER  CAN  ANALYZE  STATISTICALLY 


Section  6.  Taxonomies  of  Tasks  and  Associated  Skill  Levels 

Circle  the  scale  number  that  best  indicates  your  degree  of  response  to  each  item.  Put 
an  "X"  in  the  "Don't  Know"  box  for  items  to  which  you  cannot  respond. 

6-1.  Considering  my  present  job  as  a  designer,  the  information  in  Section  6  is: 

| - I - 1 - 1 - 1 - ? - 1 - 4 - 1 - 5 - 1  [  ] 

Very  Completely  Don't 

Useful  Useless  Know 


6-2.  If  DOD  were  to  seriously  impose  design  constraints  based  on  manpower  considera¬ 
tions,  the  information  in  Section  6  would  be: 


1  1  1 

2 

I  3 

|  4.._ 

1  5  -1 

[  ] 

1  r 

Very 
Useful 

1 

1 

Completely 

Useless 

Don 1 1 
Know 

6-3. 

The  information  in  Section  6  appears  to  be: 

■  1  ,  2  ,  3  ,  4 

|  5  . 

[  1 

1  !  1  1 

Very 

1  ! 

Very 

Don't 

Accurate 

Questionable 

Know 

6-4. 

The  information  in  Section  6  is: 

L_J _ L  ^  I 

3  , 

1  4  1  5  ! 

[  ] 

1  1  1 

Thorough 

i  1  1 

Full  of  Gaps 

Don't 

&  Complete 

and  Omissions 

Know 

6-5. 

The  format  (tables,  charts,  text,  etc.) 
in  Section  6  is: 

1 _ ] _ L  ^  ■  3 

for  presenting  the  key  information 

_  i  4  .  5,  [  ] 

1  .1  1 

Appropriate; 

i  i  i 

Inappropriate; 

Don't 

Easy  to 

Hard  to 

Know 

Understand 

Understand 

6-6. 

In  order  to  provide  helpful  guidance  to  designers  like  me.  Section 
be: 

l  1  l _ ? _ i  3 _ 1 _ 4 _ | _ 5 _ | 

6  should 

[  3 

i  i  l 

Kept  exactly 

i  i  n 

Changed 

Don't 

as  is 

Drastically 

Know 

(or  Omitted) 
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USE  THIS  PAGE  TO  PROVIDE  EXPLANATIONS  THAT  OUR  HUMAN 
ANALYST  CAN  USE  TO  MODIFY  AND  IMPROVE  THE  GUIDE 

Section  6.  Taxonomies  of  Tasks  and  Associated  Skill  Levels 

Explain  your  reasons  for  circling  a  "4“  or  "5"  on  any  item  on  the  opposite  page. 


6-1. 


6-2. 


6-3. 


6-4. 


6-5. 


6-6. 


Use  additional  sheets  for  long  explanations.  Remember  to  indicate  the  item  number 


USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTER  CAN  ANALYZE  STATISTICALLY 


Section  7.  Difficult  and  Time-Consuming  Tasks 

Circle  the  scale  number  that  best  indicates  your  degree  of  response  to  each  item.  Put 
an  "X"  in  the  "Don’t  Know”  box  for  items  to  which  you  cannot  respond. 

7-1.  Considering  ny  present  job  as  a  designer,  the  information  in  Section  7  is: 

i — '■ — i — - — t — 5 — i — - — i — 5 — i 

Very  Completely 

Useful  Useless 


[  ] 

Don’t 

Know 


7-2. 


7-3. 


7-5. 


7-6. 


If  DOD  were  to  3ez*iouslu  impose  design  constraints  based  on  manpower  considera¬ 
tions,  the  information  in  Section  7  would  be: 


1  1  ,  2 

1  3 

,  4 

1  _  5 _ , 

[  ] 

1 -  1 

Very 

I 

I 

1 

Completely 

Don't 

Useful 

Useless 

Know 

The  information  in  Section  7 

appears  to 

be: 

,1,2 

!  3 

i  4 

,  .  5  , 

[  ] 

i  i 

Very 

i 

1 

\  i 

Very 

Don't 

Accurate 

Questionable 

Know 

The  information  in  Section  7 

is: 

|  1  t  2 

,  3 

1  4 

1  5  ! 

[  ] 

1  ! 
Thorough 

l 

1 

1  1 

Full  of  Gaps 

Don’t 

&  Complete 

and  Omissions 

Know 

The  format  (tables,  charts. 

text,  etc.) 

for  presenting  the  key  information 

in  Section  7  is: 

1  1  -1  2 

!  1  3 

1  4 

1  5  1 

[  1 

1  t 

Appropriate; 

1 

1 

1  1 

Inappropriate; 

Don’t 

Easy  to 

Hard  to 

Know 

Understand 

Understand 

In  order  to  provide  helpful 

guidance  to  designers 

like  me.  Section 

7  should 

be: 

1  1  1  2 

1  3 

(  4 

|  5  ■ 

[  1 

Kept  exactly 

1 

1 

1  1 

Changed 

Don't 

as  is 

Drastically 

Know 

(or  Omitted) 
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USE  THIS  PAGE  TO  PROVIDE  EXPLANATIONS  THAT  OUR  HUMAN 
ANALYST  CAN  USE  TO  MODIFY  AND  IMPROVE  THE  GUIDE 


Section  7.  Difficult  and  Time-Consuming  Tasks 

Explain  your  reasons  for  circling  a  "4"  or  "5"  on  any  item  on  the  opposite  page. 


7-1. 


USE  THIS  PAGE  TO  PROVIDE  STRUCTURED  RESPONSES  THAT 
OUR  COMPUTER  CAN  ANALYZE  STATISTICALLY 


Section  8.  Training  Requirements  and  NECs 

Circle  the  scale  number  that  best  indicates  your  decree  of  response  to  each  item.  Put 
an  "X"  in  the  "Don't  Know"  box  for  items  to  which  you  cannot  respond. 


8-1.  Cor.s'der-'r.g  my  present  job  as  a  designer,  the  information  in  Section  8  is: 

,  1  .2,3  ,4 _ ,  5 _  ,  [  ] 


Very 

Useful 


Completely 

Useless 


Don't 

Know 


If  DOD  were  to  seviousli;  impose  design  constraints  based  on  manpower  considera¬ 
tions,  the  information  in  Section  8  would  be: 

I _ 1 _ t _ 1 _ u-J - 1 - ? - ( - 5 - j  t  1 

Very  Completely  Don’t 

Useful  Useless  Know 


The  information  in  Section  8  appears  to  be: 

| _ 1 _ I _ 1 _ (_? _ j- 

Very 

Accurate 


Very  Don't 

Questionable  Know 


The  information  in  Section  8  is: 

,1,2, 

I - ? - T 

Thorough 
&  Complete 


Full  of  Gaps 
and  Omissions 


Don't 

Know 


The  format  (tables,  charts,  text,  etc.)  for  presenting  the  key  information 
in  Section  8  is: 

! _ 1 _ | _ l _ | _ 3 _ | _ 5 _ | _ 3 _ i  1  ] 

Appropriate;  Inappropriate;  ^on 

Easy  to  Hard  to  Know 

Understand  Understand 


In  order  to  provide  helpful  guidance  to  designers  like  me.  Section  8  should 
be: 


Kept  exactly 
as  is 


Changed 
Drastically 
(or  Omitted) 


Don't 

Know 


USE  THIS  PAGE  TO  PROVIDE  EXPLANATIONS  THAT  OUR  HUMAN 
ANALYST  CAN  USE  TO  MODIFY  AND  IMPROVE  THE  GUIDE 


Section  8.  Training  Requirements  and  NECs 

Explain  your  reasons  for  circling  a  "4"  or  "5"  on  any  item  on  the  opposite  page. 


Use  additional  sheets  for  long  explanations.  Remember  to  indicate  the  item  number. 

A-29 
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Section  9.  Billet  Lifecycle  Costs  for  Required  Personnel 

Circle  th  ■  scale  number  that  bees  indicates  your  degree  of  response  to  each  item.  Put 
an  "X"  in  the  "Don't  Know"  box  for  items  to  which  you  cannot  respond. 
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Section  10.  References 
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PLEASE  FILL  IN  THE  BACKGROUND  INFORMATION  REQUESTED  BELOW. 
PLEASE  PRINT  CAREFULLY  A  FULL  MAILING  ADDRESS  TO  ENSURE  RECEIPT  OF 
THE  HONORARIUM. 

IAME  .  _ PHONE  (  ) 


COMPANY 


DIVISION 


JOB  TITLE 


BRIEF  JOB  DESCRIPTION 


1.  What  system  development  programs  have  you  worked  with  in  the  last  5  years? 


System  Designator  Type  of  System 


Your  Major  Responsibilities 


2.  Approximately  how  much  time  did  you  spend  in  examining  the  Guide? 

_ hours  _ minutes 

3.  Approximately  how  much  time  did  you  spend  in  filling  out  this  evaluation  package? 

_ hours  _ minutes 

4.  Honorarium  mailing  address  _ _________ 


5.  Thank  you  for  spending  your  valuable  time  and  energy  in  helping  us. 


APPENDIX  B 

PERSONS  WHO  MADE  SUBSTANTIVE  CONTRIBUTIONS  TO  THE  EVALUATION 
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Persons  Who  Made  Substantia]  Contributions  to  the  Evaluation 

DoD  Personnel 

CNET  -  McCluskey  (Program  Analyst)  -  Code  N315.  (9  July  1980,  Evaluation  Package  G) 

NAVELEX  -  Brooks  (Branch  Head),  Golden,  and  Blankenheim  -  ELEX  4701  -  System 
Effectiveness  Engineering  Division  -  Maintainability  and  Human  Engineering  Branch. 
Privateer  (Director)  -  ELEX  460  -  Maintenance  and  Material  Support  Division 

NAVELEX  -  Hobart  (Technical  Director)  -  ELEX  05A  -  Materiel  Acquisition  Directorate. 
(9  April  1980,  Memo) 

NAVELEX  -  Holmes  (Director)  -  ELEX  310  -  Research  and  Technology  Directorate  - 
Telecommunications  Division.  (28  November  1979,  Memo) 

NAVELEX  -  Miller  (Branch  Head)  -  ELEX  5203  -  Air  Traffic  Control  and  Navigation 
Systems  Division  -  Ships  System  Applications  Branch.  Bell  Aerospace  -  Junke  (Project 
Manager)  and  Femiano  (Technical  Director):  SPN-42  Surface  ship  radar.  (22  January 
1980,  Group  Interview) 

NAVELEX  -  5pangler  (Technical  Director)  -  ELEX  501 A  -  Material  Acquisition 
Directorate  -Ship  Programs  and  Systems  Integration  Division.  (November  1979, 
Telephone  Interview) 

NAVELEX  -  Von  Perbandt  (Deputy  Commander)  -  ELEX  04  -  Logistics  Directorate.  (24 
April  1980,  Memo) 

NAV5EA  -  Lee  (Branch  Head)  -  SEA  05L1C11  -  PATAO  -  Strike  Warfare  Program  - 
Combat  Systems  Branch.  (2  January  1980,  Information  Feedback  Sheet;  23  April  1980, 
Telephone  Interview) 

NAVSEA  -  Lewis  (Logistics  Management  Specialist)  -  PMS  300C  -  Combatant 
Craft/Service  Craft/ Amphib  Acquisition  Projects  -  Program  Management.  (11 
December  1979,  Telephone  Interview;  18  April  1980,  Telephone  Interview) 

NAVSEA  -  Mellis  (Analyst)  -  SEA  313  -  Ships  Concept  and  Development  Group  Systems 
Engineering  Division  -  Manning  and  Controls  Integration  Branch.  (20  November  1979, 
Telephone  Interview) 

NAVSEA  -  Plato  (Branch  Head),  Luce,  Mellis,  Stein,  Carlson,  and  Judge  -  SEA  313  -  Ships 
Concepts  and  Development  Group  -  Systems  Engineering  Division  -  Manning  and 
Controls  Integration  Branch.  (21  January  1980,  Group  Interview) 

NAVSEA  -  Vroom  (Engineering  Research  Psychologist)  -  SEA  05L1C1  -PATAO  -  Strike 
Warfare  Program  -  Strike  Warfare.  (January  1980,  Personal  Notes;  29  April  1980, 
Telephone  Interview;  27  May  1980,  Evaluation  Package  G  and  Letter) 

NAVSEA  -  Walker  (Engineering  Psychologist)  -  SEA  05L1C31  -  PATAO  -  ASW/Ship 
Support  Program.  (15  January  1980,  Memo;  June  1980,  Evaluation  Package  G) 

NAVSEA  -  Young  (Manpower  and  Training  Analyst)  -  SEA  05L1C41  -  PATAO  -  Special 
Systems  Program,  (January  1980,  Memo) 


NWF.SA  -  Casey  (Analyst)  -  ESA  834.  (November  1979.  Information  Feedback  Sheet) 

NESSEC  -  Dwinelle  (Analyst)  -  Code  2102.  (16  January  1980,  Information  Feedback  Sheet) 

NOSC  -  Nunn  (Division  Head)  -  Code  921  -  Test  Technology  Office.  (13  February  1980, 
Interview) 

NOSC  -  Ward  (Division  Head)  and  Townsend  -  Code  924  -  Electronic  Engineering  and 
Services  Department  -  Design  Engineering  Division.  (January  1980,  Memo;  13  February 
1980,  Group  Interview) 

OPNAV  -  Beggs  -  OP  321 D  -  Surface  Warfare  -  Surface  Combatants/Modernization/ 

Readiness  Ship  Maintenance/Modernization.  (January  1980,  Information  Feedback 
Sheet) 

Contractor  Personnel 

Bendix  -  Chief  Engineer,  Design:  CAPTOR,  air  or  surface  implanted  mine  system;  AQS- 
17,  airborne  mine  relocating  sonar.  (16  May  1980,  Evaluation  Package  C) 

Bendix  -  Chief  Engineer,  Program  Manager:  AQS-13E,  airborne  sonar  signal  processor; 
AQS-18.  helicopter  dipping  sonar.  (12  May  1980,  Letter;  16  May  1980,  Evaluation 
Package  C) 

Bendix  -  Program  Manager:  SEAGUARD,  deep  water  acoustic  system;  AOS- 13,  airborne 
sonar;  F-14,  airplane,  primary  hydraulic  system.  (16  May  1980,  Evaluation  Package  C) 

FMC  -  Project  Engineer,  Design:  GMLS  MK  13/4,  TARTAR  missile  launcher  for  FFG-7; 
GMLS  MK  26,  TARTAR/AEGIS  missile  launcher  for  CGN-38, 

DD-993,  and  CG-47;  EX  72,  interface  simulator  for  AEGIS  GMLS.  (6  June  1980, 
Evaluation  Package  C) 

FMC  -  Senior  Project  Engineer:  MIU  (Message  Interface  Unit),  missile  setting  interface; 
digital  data  bus,  surface  shipboard  interior  communications;  gunmount  and  missile 
launcher  components;  microprocessor-based  test  equipment.  (6  June  1980,  Evaluation 
Package  C) 

FMC  -  Project  Leader,  Design:  MK  71/0  surface  ship  8"  gunmount;  MK  45/0,  surface  ship 
5"  gunmount;  MK  42/11,  surface  ship  5"  gunmount;  GMLS  MK  13/4,  TARTAR  missile 
launching  system.  (6  June  1980,  Evaluation  Package  C) 

FMC  -  Design  Assurance  Manager:  GMLS  MK  26,  missile  launching  system;  GMLS 
MK  13/4,  missile  launching  system.  (6  June  1980,  Evaluation  Package  C) 

FMC  -  Chief  Engineer,  Electrical:  GMLS  MK  26,  missile  launching  system;  GMLS  MK 
13/4,  missile  launching  system;  MK  45,  5"  surface  ship  gunmount;  MK  42/11,  surface 
ship  5"  gunmount  upgrade;  GMLS  MK  10,  missile  launching  system.  (5  May  1980, 
Telephone  Interview;  6  June  1980,  Evaluation  Package  C;  6  June  1980,  Letter) 

GE  -  Systems  Engineer:  SQR-19  (TACTAS),  surface  ship  tactical  sonar  system;  SURTAS, 
surface  ship  towed  array  surveillance  sonar.  (16  June  1980,  Evaluation  Package  C) 

GE  -  Program  Manager:  SOS-26,  surface  ship  sonar;  SQS-53,  surface  ship  sonar;  SQQ-89, 
sonar  display;  SQR-19,  surface  ship  sonar;  ASWCSI  control  system.  (13  June  1980, 
Telephone  Interview;  16  June  1980,  Evaluation  Package  C;  16  June  1980,  Letter) 


GE  -  Manager,  Systems  Engineering  and  Implementation:  SQR-19,  surface  ship  towed 
array  sonar;  SQS-53,  surface  ship  sonar;  COBRA  JUDY;  site  defense  radar.  (16  June 
1980,  Evaluation  Package  C) 

GE  -  Manager,  Design  Review:  SQS-53  PEC,  surface  ship  sonar.  (16  June  1980, 
Evaluation  Package  C) 

GE  -  Manager,  Systems  Engineering:  SQR-19,  surface  ship  towed  array  sonar.  (16  June 
1980,  Evaluation  Package  C). 

Goodyear  -  Program  Manager,  Engineering:  MK  48,  wire  guided  torpedo,  surface  or 
submarine;  TRIDENT  submarine,  ground  support;  CAPTOR,  implanted  mine,  air  or 
surface.  (12  June  1980,  Evaluation  Package  C) 

Goodyear  -  Manager,  Product  Engineering:  PERSHING  II,  terminal  guidance,  army 
missile;  SUBROC,  submarine  fire  control  and  guidance  system.  (12  June  1980, 
Evaluation  Package  C) 

Goodyear  -  Manager,  Product  Engineering:  F15  simulator;  P  2,  guidance  system.  (12  June 
1980,  Evaluation  Package  C) 

Gould  -  Manager,  Electrical  Engineering:  TTUMS,  towed  array  measurement  system; 
DDCS,  seismic  data  collection  system.  (25  June  1980,  Evaluation  Package  C) 

Gould  -  Lead  Engineer:  DDCS,  seismic  data  collection  system;  Vickers,  remote  control 
system.  (16  June  1980,  Evaluation  Package  C) 

Hughes  -  Senior  Systems  Engineer:  NP/CSD,  large  screen  display,  air  system;  SSN  display 
suite;  MK  113/10,  submarine  fire  control  system;  BQQ-5,  submarine  sonar  system.  (23 
June  1980,  Evaluation  Package  C) 

Hughes  -  Program  -Manager;  Senior  Design  Engineer;  Senior  Engineer;  and  Design 
Engineer:  SURTAS,  surface  ship  sonar;  BQQ-5,  submarine  sonar;  MK  112,  submarine 
fire  control  system;  SQS-53,  surface  ship  sonar;  SQR-19,  surface  ship  sonar.  (11  June 
1980,  Group  Interview) 

ITT  -  Gilfillan  -  Project  Engineer,  Design:  5PS-48C,  surface  ship  3-D  radar.  (1  May  1980, 
Evaluation  Package  C) 

ITT  -  Gilfillan  -  Manager,  Product  Engineering:  SPS-48,  surface  ship  3-D  radar;  TPN-22, 
approach  control  radar,  Marine  Corps;  TPN-18,  air  traffic  control  radar,  Marine  Corps; 
FPN-40,  air  traffic  control  radar,  Marine  Corps.  (1  May  1980,  Evaluation  Package  C) 

ITT  -  Gilfillan  -  Logistics  System  Specialist  Engineer:  SPS-48C,  surface  ship  3-D  radar; 
SPS-48  NTU,  New  Threat  Update.  (1  May  1980,  Evaluation  Package  C) 

ITT  -  Gilfillan  -  Logistics  Project  Engineer:  SP5-48C,  surface  ship  3-D  radar.  (1  May 
1980,  Evaluation  Package  C) 

ITT  -  Gilfillan  -  Senior  Systems  Engineer.  (1  May  1980,  Letter) 

Litton  -  Program  Manager:  UXC-4,  tactical /digital  facsimile  transceiver;  CSTOT, 
combat  system  team  operational  trainer,  surface/helicopter;  CAORF,  ships  bridge 
simulator;  AXR-15,  low  light  level  TV  for  5-3  Aircraft.  (29  May  1980,  Evaluation 
Package  C) 
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Litton  -  Manager,  Advanced  Systems;  UXC-4,  tactical  digital  facsimile  transceiver; 
FAST  FAX  6000,  high  speed  secure  facsimile  network;  Overlay  Generator,  multicolor 
transparency  facsimile  printer;  MASS  TAPE,  1  trillion  bit  memory  system.  (29  May 
1980,  Evaluation  Package  C) 

Lockheed  -  Project  Leader:  HIGH  ROY,  real  time  sensor  control  and  readout;  OUTLAW 
HAW'K,  command  and  control.  (27  June  1980,  Evaluation  Package  C) 

Lockheed  -  Staff  Engineer:  USO-81,  command  and  control  tactical  data  display.  (27  June 
1980,  Evaluation  Package  C) 

Lockheed  -  Project  Engineer,  R<5cD  and  ORDALTS:  MK  86,  gunfire  control  system.  (20 
May  1980,  Evaluation  Package  C) 

Lockheed  -  Chief  Engineer:  MK  86,  gunfire  control  system.  (20  May  1980,  Evaluation 
Package  C) 

Lockheed  -  Systems  Engineer:  SSURnS/DDGX,  sensor/weapons  system,  surface  ships; 
SEAFIRE,  electro-optical  sensor  system,  surface  fire  control  system;  SIRCS,  inter¬ 
mediate  range  combat  system.  (20  May  1980.  Evaluation  Package  C) 

RCA  -  Systems  Engineer,  Adequacy:  AEGIS,  combat  system.  (30  May  1980,  Evaluation 
Package  C) 

RCA  -  Manager,  Systems  Support  Engineering:  AEGIS  combat  system  and  weapon  system. 
(30  May  1980,  Evaluation  Package  C) 

RCA  -  Human  Factors  Engineer:  AEGIS  combat  system;  IRR,  Trident  communications 
trainer;  NOSS,  ocean  satellite  system.  (30  May  1980,  Evaluation  Package  C) 

RCA  -  Senior  Systems  Engineer:  AEGIS,  computer  controlled  phased  array  radar.  (30 
May  1980,  Evaluation  Package  C) 

RCA  -  Manager,  Readiness  Engineering:  AEGIS,  total  ship,  combat  system,  weapon 
system.  (30  May  1980,  Evaluation  Package  C) 

RCA  -  Manager,  Combat  System  Readiness:  AEGIS,  weapons/combat  system,  CC.-47  class 
ships.  (30  May  1980,  Evaluation  Package  C) 

RCA  -  Systems  Discipline  Engineer:  SP1-A,  radar.  (30  May  1980,  Evaluation  Package  C) 

Singer  -  Systems  Design  Engineer:  FCS  MK  117,  submarine  fire  control  system;  FCS  MK 
1 18,  Trident  submarine  fire  control  system.  (28  May  1980,  Evaluation  Package  C) 

Singer  -  Manager,  Product  Design  Department:  MK  116/1,  surface  ship  ASW  fire  control 
system;  SFCS  MK  1/0,  submarine  fire  control  system,  Australia;  MK  377/0;  power  drive 
amplifier,  ASROC  launcher.  (28  May  1980,  Evaluation  Package  C) 

Singer  -  Publications  Manager:  FCS  MK  1 13,  submarine  fire  control  system;  FCS  MK  1 16. 
surface  fire  control  system.  (28  May  1980,  Evaluation  Package  C) 

Sperry  -  Senior  Engineer:  S*3(),  ASW  aircraft  update;  Life  Cycle  Software  Support 
Facility,  large  scale  avionics  software  maintenance  facility;  VSM-247,  automatic  test 
equipment  system.  (9  June  1980,  Evaluation  Package  C) 


Sperry  -  Manager,  Test  Equipment  and  Systems  Engineering:  MK  68,  surface  ship  gunfire 
control  system;  TALOS,  surface  ship  fire  control  system.  (9  June  1980,  Evaluation 
Package  C) 


RECOMMENDED  PRELIMINARY  OUTLINE  FOR  REVISED  GUIDE 

The  guide  should  be  published  in  separate  volumes  for  each  type  of  system.  The 
following  outline  is  for  the  volume  on  surface  ship  sonar,  and  exemplifies  the  topics  to  be 
covered  for  each  type  of  system. 
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115)  (2),  (OP-39),  (OP-964D),  (OP-964D1),  (OP-987H) 

Chief  of  Naval  Material  (NMAT  00),  (NMAT  04),  (NMAT  08B),  (NMAT  08D2),  (Director, 
Strategic  System  Projects  (SP-15) 

Chief  of  Naval  Research  (Code  450)  (3),  (Code  452),  (Code  455),  (Code  458)  (2) 

Chief  of  Information  (01-2252) 

Chief  of  Naval  Technical  Training  (017) 

Director  of  Navy  Laboratories 
Commandant  of  the  Marine  Corps  (MP1-20) 

Commander  Naval  Air  Development  Center  (Code  602),  (Code  2023) 

Commander  Naval  Data  Automation  Command  (Library) 

Commander  Naval  Electronic  System  Command  (NELEX-01),  (NELEX-04C),  (NELEX- 
05A),  (NELEX-330),  (NELEX-350),  (NELEX-460),  (NELEX-4701)  (2),  (NELEX-501), 
(NELEX-504B),  (PME-107-5),  (PME-108-1) 

Commander  Naval  Military  Personnel  Command  (NMPC-013C),  (NMPC-5) 

Commander  Naval  Oc'"  _»n  Systems  Center  (Code  161),  (Code  162),  (Code  823),  (Code  8231), 
(Code  8254),  (Code  8302),  (Code  921),  (Code  9242) 

Commander  Naval  Sea  System  Command  (NSEA  05L1C),  (NSEA-05L1C1),  (NSEA- 
05L1C31),  (NSEA-3134),  (NSEA-61R2),  (NSEA-62R),  (NSEA-63R),  (PMS- 377213) 
Commander  Naval  Surface  Weapons  Center,  Dahlgren  (Code  G31) 

Commander  Naval  Weapons  Center,  China  Lake  (Code  081),  (Code  3644) 

Commander  Training  Command,  U.S.  Atlantic  Fleet  (Code  N3A) 

Commander  Training  Command,  U.S.  Pacific  Fleet 

Commanding  Officer,  Fleet  Anti-Submarine  Warfare  Training  Center,  Pacific 
Commanding  Officer,  Fleet  Combat  Training  Center,  Atlantic  (Code  02) 

Commanding  Officer,  Fleet  Combat  Training  Center,  Pacific  (Code  00E) 

Commanding  Officer,  Fleet  Training  Center,  San  Diego 

Commanding  Officer  Naval  Aerospace  Medical  Institute  (Library  Code  12)  (2) 

Commanding  Officer,  Naval  Education  and  Training  Program  Development  Center  (Tech¬ 
nical  Library)  (2) 

Commanding  Officer,  Naval  Electronic  Systems  Security  Engineering  Center  (Security 
Station),  (Code  240) 

Commanding  Officer,  Naval  Technical  Training  Center  (Code  01E) 

Commanding  Officer,  Naval  Training  Equipment  Center  (Technical  Library) 

Commanding  Off icer,  Shore  Intermediate  Maintenance  Activity  (0120) 

Director,  Naval  Education  and  Training  Program  Development  Center  Detachment 
Director,  Training  Analysis  and  Evaluation  Group  (TAEG) 

President,  Naval  War  College 

Superintendent,  Naval  Postgraduate  School 

Fleet  Master  Chief,  Naval  Material  Command  (NMAT  00C) 

Fleet  Master  Chief,  U.S.  Atlantic  Fleet  (Code  003) 

Fleet  Master  Chief,  U.S.  Pacific  Fleet 

Fleet  Master  Chief,  Chief  of  Naval  Education  and  Training  (Code  003) 

Commanding  Officer,  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences, 
Alexandria  (Reference  Service) 

Director,  U.S.  Army  TRADOC  Systems  Analysis  Activity,  White  Sands  Missile  Range 
(Library) 

Chief,  Army  Research  Institute  Field  Unit--USAREUR  (Library) 

Commander,  Air  Force  Human  Resources  Laboratory,  Brooks  Air  Force  Base  (Manpower 
and  Personnel  Division),  (Scientific  and  Technical  Information  Office) 


Commander,  Air  Force  Human  Resources  Laboratory,  Lowry  Air  Force  Base  (Technical 
Training  Branch) 

Commander,  Air  Forace  Human  Resources  Laboratory,  Williams  Air  Force  Base  (Opera¬ 
tions  Training  Division) 

Commander,  Air  Force  Human  Resources  Laboratory,  Wright-Patterson  Air  Force  Base 
(Advanced  Systems  Division),  (Logistics  and  Technical  Training  Division) 

Program  Manager,  Life  Sciences  Directorate,  Bolling  Air  Force  Base 
Commandant,  Coast  Guard  Headquarters  (G-P-l/62) 

Commanding  Officer,  U.S.  Coast  Guard  Institute 

Commanding  Officer,  U.S.  Coast  Guard  Research  and  Development  Center,  Avery  Point 
Commandant  Industrial  College  of  the  Armed  Forces 
Director,  Science  and  Technology,  Library  of  Congress 
Defense  Technical  Information  Center  (12) 


